Jan J. Koenderink

The structure of images

In practice the relevant details of images exist only over a restricted range of scale. Hence it is important to study the dependence of image structure on the level of resolution. It seems clear enough that visual perception treats images on several levels of resolution simultaneously and that this fact must be important for the study of perception. However, no applicable mathematically formulated theory to deal with such problems appers to exist. In this paper it is shown that any image can be embedded in a one-parameter family of derived images (with resolution as the parameter) in essentially only one unique way if the constraint that no spurious detail should be generated when the resolution is diminished, is applied. The structure of this family is governed by the well known diffusion equation (a parabolic, linear, partial differential equation of the second order). As such the structure fits into existing theories that treat the front end of the visual system as a continuous tack of homogeneous layer, characterized by iterated local processing schemes. When resolution is decreased the images becomes less articulated because the extrem (“light and dark blobs”) disappear one after the other. This erosion of structure is a simple process that is similar in every case. As a result any image can be described as a juxtaposed and nested set of light and dark blobs, wherein each blod has a limited range of resolution in which it manifests itself. The structure of the family of derived images permits a derivation of the sampling density required to sample the image at multiple scales of resolution. The natural scale along the resolution axis (leading to an informationally uniform sampling density) is logarithmic, thus the structure is apt for the description of size invariances.

Reflectance and texture of real-world surfaces

In this work, we investigate the visual appearance of real-world surfaces and the dependence of appearance on the geometry of imaging conditions. We discuss a new texture representation called the BTF (bidirectional texture function) which captures the variation in texture with illumination and viewing direction. We present a BTF database with image textures from over 60 different samples, each observed with over 200 different combinations of viewing and illumination directions. We describe the methods involved in collecting the database as well as the importqance and uniqueness of this database for computer graphics. A related quantity to the BTF is the familiar BRDF (bidirectional reflectance distribution function). The measurement methods involved in the BTF database are conducive to simultaneous measurement of the BRDF. Accordingly, we also present a BRDF database with reflectance measurements for over 60 different samples, each observed with over 200 different combinations of viewing and illumination directions. Both of these unique databases are publicly available and have important implications for computer graphics.

Surface shape and curvature scales

Abstract The classical surface curvature measures, such as the Gaussian and the mean curvature at a point of a surface, are not very indicative of local shape. The two principal curvatures (taken as a pair) are more informative, but one would prefer a single shape indicator rather than a pair of numbers. Moreover, the shape indicator should preferably be independent of the size i.e. the amount of curvature, as distinct from the type of curvature. We propose two novel measures of local shape, the ‘curvedness’ and the ‘shape index’. The curvedness is a positive number that specifies the amount of curvature, whereas the shape index is a number in the range [−1, +1] and is scale invariant. The shape index captures the intuitive notion of ‘local shape’ particularly well. The shape index can be mapped upon an intuitively natural colour scale. Two complementary shapes (like stamp and mould) map to complementary hues. The symmetrical saddle (which is very special because it is self-complementary) maps to white. When a surface is tinted according to this colour scheme, this induces an immediate perceptual segmentation of convex, concave, and hyperbolic areas. We propose it as a useful tool in graphics representation of 3D shape.

Affine structure from motion

A mobile observer samples sequences of narrow-field projections of configurations in ambient space. The so-called structure-from-motion problem is to infer the structure of these spatial configurations from the sequence of projections. For rigid transformations, a unique metrical reconstruction is known to be possible from three orthographic views of four points. However, human observers seem able to obtain much shape information from a mere pair of views, as is evident in the case of binocular stereo. Moreover, human observers seem to find little use for the information provided by additional views, even though some improvement certainly occurs. The rigidity requirement in its strict form is also relaxed. We indicate how solutions of the structure-from-motion problem can be stratified in such a way that one explicitly knows at which stages various a priori assumptions enter and specific geometrical expertise is required. An affine stage is identified at which only smooth deformation is assumed (thus no rigidity constraint is involved) and no metrical concepts are required. This stage allows one to find the spatial configuration (modulo an affinity) from two views. The addition of metrical methods allows one to find shape from two views, modulo a relief transformation (depth scaling and shear). The addition of a third view then merely serves to settle the calibration. Results of a numerical experiment are discussed.

Representation of local geometry in the visual system

It is shown that a convolution with certain reasonable receptive field (RF) profiles yields the exact partial derivatives of the retinal illuminance blurred to a specified degree. Arbitrary concatenations of such RF profiles yield again similar ones of higher order and for a greater degree of blurring.By replacing the illuminance with its third order jet extension we obtain position dependent geometries. It is shown how such a representation can function as the substrate for “point processors” computing geometrical features such as edge curvature. We obtain a clear dichotomy between local and multilocal visual routines. The terms of the truncated Taylor series representing the jets are partial derivatives whose corresponding RF profiles closely mimic the well known units in the primary visual cortex. Hence this description provides a novel means to understand and classify these units.Taking the receptive field outputs as the basic input data one may devise visual routines that compute geometric features on the basis of standard differential geometry exploiting the equivalence with the local jets (partial derivatives with respect to the space coordinates).

What Does the Occluding Contour Tell Us about Solid Shape

A new theorem is discussed that relates the apparent curvature of the occluding contour of a visual shape to the intrinsic curvature of the surface and the radial curvature. This theorem allows the formulation of general laws for the apparent curvature, independent of viewing distance and regardless of the fact that the rim (the boundary between the visible and invisible parts of the object) is a general, thus twisted, space curve. Consequently convexities, concavities, or inflextions of contours in the retinal image allow the observer to draw inferences about local surface geometry with certainty. These results appear to be counterintuitive, witness to the treatment of the problem by recent authors. It is demonstrated how well-known examples, used to show how concavities and convexities of the contour have no obvious relation to solid shape, are actually good illustrations of the fact that convexities are due to local ovoid shapes, concavities to local saddle shapes.

Facts on optic flow

We employ an optimal solution to both the “shape from motion problem” and the related problem of the estimation of self-movement on a purely optical basis to deduce practical rules of thumb for the limits of the optic flow information content in the presence of perturbation of the motion parallax field. The results are illustrated and verified by means of a computer simulation.The results allow estimates of the accuracy of depth and egomotion estimates as a function of the accuracy of data sampling and the width of field of view, as well as estimates of the interaction between rotational and translational components of the movement.

Scale and the differential structure of images

Why and how one should study a scale-space is prescribed by the universal physical law of scale invariance, expressed by the so-called Pi-theorem. The fact that any image is a physical observable with an inner and outer scale bound, necessarily gives rise to a ‘scale-space representation’, in which a given image is represented by a one-dimensional family of images representing that image on various levels of inner spatial scale. An early vision system is completely ignorant of the geometry of its input. Its primary task is to establish this geometry at any available scale. The absence of geometrical knowledge poses additional constraints on the construction of a scale-space, notably linearity, spatial shift invariance and isotropy, thereby defining a complete hierarchical family of scaled partial differential operators: the Gaussian kernel (the lowest order, resettling operator) and its linear partial derivatives. They enable local image analysis through the detection of local differential structure in a robust way, while at the same time capturing global features through the extra scale degree of freedom. In this paper we show why the operations of scaling and differentiation cannot be separated. This framework permits us to construct in a systematic way multiscale, cartesian differential invariants, i.e. true image descriptors that exhibit manifest invariance with respect to a change of cartesian coordinates. The scale-space operators closely resemble the receptive field profiles found in mammalian frontend visual systems.

The Singularities of the Visual Mapping

In this article we treat purely metrical properties of the visual image, e.g. the time changes of the relative positions and orientations of image details. Self-induced movements of an observer relative to rigid bodies in his environment generate charactertistic motion parallax fields. The observer may regard those fields as proprioceptive and interprete the geometrical invariants of the fields as indicators of solid shape. In this way his perceptions become object-oriented, which is the normal case as the many constancy-phenomena show. Similar arguments apply to the disparity field of binocular vision. In this paper we treat the qualitative nature of such fields. [In this case the qualitative nature is basic. Compare the case of an equation with a single unknown. Often one is interested primarily in the qualitative solution (are there roots? How many?), and only slightly in the quantitative information (the numerical value of a root).] The qualitative nature of the fields is fixed if their singularities are known. It is shown that the singularities are of two types: isolated points (so-called specular points) and line-singularities (so-called folds, cusps and T-junctions). It is shown that for most vantage points that an observer can occupy, the topological structure of the set of singularities does not change if the observer performs small exploratory movements. That is most vantage points are stable. At an unstable vantage point the set of singularities changes and the observer experiences an event. Because certain properties of the set of singularities are shown to be preserved, only a few simple types of event are possible. A complete list is presented. The occurrence of an event is shown to be simply related to the solid shape of the objects of vision. Our geometrical theory enables us to understand the structure of the observers internal models of external bodies.

Spatiotemporal Modulation Transfer in the Human Eye

The contrast sensitivity of the human eye for sinusoidal illuminance changes in space and time, obtained by means of traveling-wave stimuli, was measured as a function of spatial and temporal frequency for white light. The average retinal illuminance was varied between 0.85 and 850 trolands. The threshold modulation for perception of a moving grating is generally higher than that for detection of brightness changes, in space and/or time, that give rise to flicker phenomena. Flicker-fusion characteristics, as determined from the thresholds for the flicker phenomenon, are found to lose their band-pass-filter resemblance for spatial frequencies of more than 5 cycles per degree of visual angle. The thresholds at flicker fusion for spatial- and temporal-frequency combinations in which not both frequencies are very low, appear to be proportional to the inverse of the square root of mean retinal illuminance, in the investigated range. This suggests a photon-noise-dependent threshold mechanism which is operative in a wider illuminance range than that found with contrast-sensitivity measurements for periodic illuminance variations only in space or only in time.