J. Farley Norman

Anterior Regions of Monkey Parietal Cortex Process Visual 3D Shape

The intraparietal cortex is involved in the control of visually guided actions, like reach-to-grasp movements, which require extracting the 3D shape and position of objects from 2D retinal images. Using fMRI in behaving monkeys, we investigated the role of the intraparietal cortex in processing stereoscopic information for recovering the depth structure and the position in depth of objects. We found that while several areas (CIP, LIP, and AIP on the lateral bank; PIP and MIP on the medial bank) are activated by stereoscopic stimuli, AIP and an adjoining portion of LIP are sensitive only to depth structure. Furthermore, only these two regions are sensitive to both the depth structure and the 2D shape of small objects. These results indicate that extracting 3D spatial information from stereo involves several intraparietal areas, among which AIP and anterior LIP are more specifically engaged in extracting the 3D shape of objects.

The visual perception of smoothly curved surfaces from minimal apparent motion sequences

A series of four experiments was designed to investigate the minimal amounts of information required to perceive the structure of a smoothly curved surface from its pattern of projected motion. In Experiments 1 and 2, observers estimated the amplitudes of sinusoidally corrugated surfaces relative to their periods. Observers’ judgments varied linearly with the depicted surface amplitudes, but the amount of perceived relative depth was systematically overestimated by approximately 30%. The observers’ amplitude judgments were also influenced to a lesser extent by the amount of rotary displacement of a surface at each frame transition, and by increasing the length of the apparent motion sequences from two to eight frames. The latter effect of sequence length was quite small, however, accounting for less than 3% of the variance in the observers’ judgments. Experiments 3 and 4 examined observers’ discrimination thresholds for sinusoidally corrugated surfaces of variable amplitude and for ellipsoid surfaces of variable eccentricity. The results revealed that observers could reliably detect differences of surface structure as small as 5%. The length of the apparent motion sequences had no detectable effect on these tasks, although there were significant effects of angular displacement and surface orientation. These results are considered with respect to the analysis of affine structure from motion proposed by Todd and Bressan (1990).

The visual and haptic perception of natural object shape

In this study, we evaluated observers’ ability to compare naturally shaped three-dimensional (3-D) objects, using their senses of vision and touch. In one experiment, the observers haptically manipulated 1 object and then indicated which of 12 visible objects possessed the same shape. In the second experiment, pairs of objects were presented, and the observers indicated whether their 3-D shape was thesame ordifferent. The 2 objects were presented either unimodally (vision-vision or haptic-haptic) or cross-modally (vision-haptic or haptic-vision). In both experiments, the observers were able to compare 3-D shape across modalities with reasonably high levels of accuracy. In Experiment 1, for example, the observers’ matching performance rose to 72% correct (chance performance was 8.3%) after five experimental sessions. In Experiment 2, small (but significant) differences in performance were obtained between the unimodal vision-vision condition and the two cross-modal conditions. Taken together, the results suggest that vision and touch have functionally overlapping, but not necessarily equivalent, representations of 3-D shape.

The perception of surface orientation from multiple sources of optical information

An orientation matching task was used to evaluate observers’ sensitivity to local surface orientation at designated probe points on randomly shaped 3-D objects that were optically defined by texture, lambertian shading, or specular highlights. These surfaces could be stationary or in motion, and they could be viewed either monocularly or stereoscopically, in all possible combinations. It was found that the deformations of shading and/or highlights (either over time or between the two eyes’ views) produced levels of performance similar to those obtained for the optical deformations of textured surfaces. These findings suggest that the human visual system utilizes a much richer array of optical information to support its perception of shape than is typically appreciated.

The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure?

Three experiments are reported in which observers judged the three-dimensional (3-D) structures of virtual or real objects defined by various combinations of texture, motion, and binocular disparity under a wide variety of conditions. The tasks employed in these studies involved adjusting the depth of an object to match its width, adjusting the planes of a dihedral angle so that they appeared orthogonal, and adjusting the shape of an object so that it appeared to match another at a different viewing distance. The results obtained on all of these tasks revealed large constant errors and large individual differences among observers. There were also systematic failures of constancy over changes in viewing distance, orientation, or response task. When considered in conjunction with other, similar reports in the literature, these findings provide strong evidence that human observers do not have accurate perceptions of 3-D metric structure.

Attention to 3-D Shape, 3-D Motion, and Texture in 3-D Structure from Motion Displays

We used fMRI to directly compare the neural substrates of three-dimensional (3-D) shape and motion processing for realistic textured objects rotating in depth. Subjects made judgments about several different attributes of these objects, including 3-D shape, the 3-D motion, and the scale of surface texture. For all of these tasks, we equated visual input, motor output, and task difficulty, and we controlled for differences in spatial attention. Judgments about 3-D shape from motion involve both parietal and occipito-temporal regions. The processing of 3-D shape is associated with the analysis of 3-D motion in parietal regions and the analysis of surface texture in occipito-temporal regions, which is consistent with the different behavioral roles that are typically attributed to the dorsal and ventral processing streams.

Parietal regions processing visual 3D shape extracted from disparity

Three-dimensional (3D) shape is important for the visual control of grasping and manipulation. We used fMRI to study the processing of 3D shape extracted from disparity in human parietal cortex. Subjects stereoscopically viewed random-line stimuli portraying a 3D structure, a 2D structure in multiple depth planes or a 2D structure in the fixation plane. Subtracting the second from the first condition yields depth-structure sensitive regions and subtracting the third from the second position-in-depth sensitive regions. Two anterior intraparietal sulcus (IPS) regions, the dorsal IPS medial (DIPSM) and the dorsal IPS anterior (DIPSA) regions, were sensitive to depth structure and not to position in depth, while a posterior IPS region, the ventral IPS (VIPS) region, had a mixed sensitivity. All three IPS regions were also sensitive to 2D shape, indicating that they carry full 3D shape information. Finally DIPSM, but not DIPSA was sensitive to a saccade-related task. These results underscore the importance of anterior IPS regions in the processing of 3D shape, in agreement with their proximity to grasping-related regions. Moreover, comparison with the results of Durand, J.B., Nelissen, K., Joly, O., Wardak, C., Todd, J.T., Norman, J.F., Janssen, P., Vanduffel, W., Orban, G.A., 2007. Anterior Regions of Monkey Parietal Cortex Process Visual 3D Shape. Neuron 55, 493-505 obtained in the monkey indicates that DIPSA and DIPSM may represent human homologues for the posterior part of AIP and the adjoining part of LIP respectively.

The perceptual analysis of structure from motion for rotating objects undergoing affine stretching transformations

In two experiments, we evaluated the ability of human observers to make use of second-order temporal relations across three or more views of an apparent motion sequence for the perceptual analysis of three-dimensional form. Ratings of perceived rigidity were obtained in Experiment 1 for objects rotating in depth that were simultaneously subjected to sinusoidal affine stretching transformations along the line of sight or in a direction parallel to the image plane. Such transformations are theoretically interesting because they cannot be detected by analyses that are restricted to first-order temporal relations (i.e., two views), but they can be detected by more conventional analyses of structure from motion in which second-order temporal relations over three or more views are used. The current results show that human observers can perceive stretching transformations of a rotating 3-D object in a direction parallel to the image plane but that they fail to perceive stretching transformations along the line of sight. This result suggests that human observers can make use of some limited second-order temporal information. This finding was confirmed in Experiment 2, in which we investigated the effects of several specific optical consequences of sinusoidal stretching transformations applied in different directions. The results indicate that observers may be sensitive to the sign of acceleration, but that. they cannot make use of the precise magnitude of second-order relations necessary to recover euclidean metric structure.

Perception of Three-Dimensional Shape From Specular Highlights, Deformations of Shading, and Other Types of Visual Information:

There have been numerous computational models developed in an effort to explain how the human visual system analyzes three-dimensional (3D) surface shape from patterns of image shading, but they all share some important limitations. Models that are applicable to individual static images cannot correctly interpret regions that contain specular highlights, and those that are applicable to moving images have difficulties when a surface moves relative to its sources of illumination. Here we describe a psychophysical experiment that measured the sensitivity of human observers to small differences of 3D shape over a wide variety of conditions. The results provide clear evidence that the presence of specular highlights or the motions of a surface relative to its light source do not pose an impediment to perception, but rather, provide powerful sources of information for the perceptual analysis of 3D shape.

The detection of surface curvatures defined by optical motion

The detectability of surface curvatures defined by optical motion was evaluated in three experiments. Observers accurately detected very small amounts of curvature in a direction perpendicular to the direction of rotation, but they were less sensitive to curvatures along the direction of rotation. Variations in either the number of points (between 91 and 9) or the number of views (from 15 to 2) had little or no effect on discrimination accuracy. The results of this study demonstrate impressive visual sensitivity to surface curvature. Several characteristics of this sensitivity to curvature are inconsistent with many computational models for deriving three-dimensional structure from motion.