Jeff B. Pelz

Memory representations in natural tasks

The very limited capacity of short-term or working memory is one of the most prominent features of human cognition. Most studies have stressed delimiting the upper bounds of this memory in memorization tasks rather than the performance of everyday tasks. We designed a series of experiments to test the use of short-term memory in the course of a natural hand-eye task where subjects have the freedom to choose their own task parameters. In this case subjects choose not to operate at the maximum capacity of short-term memory but instead seek to minimize its use. In particular, reducing the instantaneous memory required to perform the task can be done by serializing the task with eye movements. These eye movements allow subjects to postpone the gathering of task-relevant information until just before it is required. The reluctance to use short-term memory can be explained if such memory is expensive to use with respect to the cost of the serializing strategy.

The coordination of eye, head, and hand movements in a natural task.

Abstract. Relatively little is known about movements of the eyes, head, and hands in natural tasks. Normal behavior requires spatial and temporal coordination of the movements in more complex circumstances than are typically studied, and usually provides the opportunity for motor planning. Previous studies of natural tasks have indicated that the parameters of eye and head movements are set by global task constraints. In this experiment, we explore the temporal coordination of eye, head, and hand movements while subjects performed a simple block-copying task. The task involved fixations to gather information about the pattern, as well as visually guided hand movements to pick up and place blocks. Subjects used rhythmic patterns of eye, head, and hand movements in a fixed temporal sequence or coordinative structure. However, the pattern varied according to the immediate task context. Coordination was maintained by delaying the hand movements until the eye was available for guiding the movement. This suggests that observers maintain coordination by setting up a temporary, task-specific synergy between the eye and hand. Head movements displayed considerable flexibility and frequently diverged from the gaze change, appearing instead to be linked to the hand trajectories. This indicates that the coordination of eye and head in gaze changes is usually the consequence of a synergistic linkage rather than an obligatory one. These temporary synergies simplify the coordination problem by reducing the number of control variables, and consequently the attentional demands, necessary for the task.

Morphological texture-based maximum-likelihood pixel classification based on local granulometric moments

Abstract Morphological granulometries are one-parameter filter sequences that monotonically decrease image area. A size distribution is generated by measuring the residual area after each iteration of the filter sequence. Normalization yields a probability distribution function whose moments can be employed as image signatures. By measuring residual area locally over a window about each point of an image instead of over the entire image, local texture features are generated at each pixel, and these features can be employed for pixel classification. By using several granulometric structuring-element generating sequences, numerous moment sets result, each carrying different textural information. A detailed analysis of this pixel classification methodology using a Gaussian maximum likelihood classifier is provided. Included is a statistical study of classification accuracy, feature optimization, and robustness with respect to various relevant noise models.

Top-down control of eye movements: Yarbus revisited

Alfred Yarbus (1967) reported that an observers eye movement record varied based on high-level task. He found that an observers eye movement patterns during freeview were dramatically different than when given tasks such as “Remember the clothes worn by the people.” Although Yarbus’ work is often cited to demonstrate the task-dependence of eye movements, it is often misrepresented; Yarbus reported results for only one observer, but authors commonly refer to Yarbus’ “observers”. Additionally, his observer viewed the painting for 21 minutes with optical stalks attached to the sclera and with his head severely restricted. Although eye movements are undoubtedly influenced by high-level tasks, it is not clear how Yarbus’ results reflect his unique experimental conditions. Because of Yarbus’ role in the literature, it is important to determine the extent to which his results represent a sample of naïve observers under more natural conditions. We replicated Yarbus’ experiment using a head-free eyetracker with 17 naïve observers. The presentations were self-paced; viewing times were typically an order of magnitude shorter than the times Yarbus imposed. Eye movement patterns were clearly task dependent, but some of the differences were much less dramatic than those shown in Yarbus’ now-classic observations.

Predictive eye movements in natural vision

In the natural world, the brain must handle inherent delays in visual processing. This is a problem particularly during dynamic tasks. A possible solution to visuo-motor delays is prediction of a future state of the environment based on the current state and properties of the environment learned from experience. Prediction is well known to occur in both saccades and pursuit movements and is likely to depend on some kind of internal visual model as the basis for this prediction. However, most evidence comes from controlled laboratory studies using simple paradigms. In this study, we examine eye movements made in the context of demanding natural behavior, while playing squash. We show that prediction is a pervasive component of gaze behavior in this context. We show in addition that these predictive movements are extraordinarily precise and operate continuously in time across multiple trajectories and multiple movements. This suggests that prediction is based on complex dynamic visual models of the way that balls move, accumulated over extensive experience. Since eye, head, arm, and body movements all co-occur, it seems likely that a common internal model of predicted visual state is shared by different effectors to allow flexible coordination patterns. It is generally agreed that internal models are responsible for predicting future sensory state for control of body movements. The present work suggests that model-based prediction is likely to be a pervasive component in natural gaze control as well.

Classroom Interpreting and Visual Information Processing in Mainstream Education for Deaf Students: Live or Memorex®?

This study examined visual information processing and learning in classrooms including both deaf and hearing students. Of particular interest were the effects on deaf students’ learning of live (three-dimensional) versus video-recorded (two-dimensional) sign language interpreting and the visual attention strategies of more and less experienced deaf signers exposed to simultaneous, multiple sources of visual information. Results from three experiments consistently indicated no differences in learning between three-dimensional and two-dimensional presentations among hearing or deaf students. Analyses of students’ allocation of visual attention and the influence of various demographic and experimental variables suggested considerable flexibility in deaf students’ receptive communication skills. Nevertheless, the findings also revealed a robust advantage in learning in favor of hearing students

Morphological image segmentation by local granulometric size distributions

Morphological granulometries are generated by successively opening a thresholded image by an increasing sequence of structuring elements. The result is a sequence of images, each of which is a subimage of the previous. By counting the number of pixels at each stage of the granulometry, a size distribution is generated that can be employed as a signature of the image. Normalization of the size distribution produces a probability distribution in the usual sense. An adaptation of the method that is appropriate to texture-based segmentation is described. Rather than construct a single size distribution based on the entire image, local size distributions are computed over windows within the image. These local size distributions lead to granulometric moments at pixels within the image, and if the image happens to be partitioned into regions of various texture, the local moments will tend to be homogeneous over any given region. Segmentation results from segmenting images whose gray values are local moments. Especially useful are the means of the local size distributions. Goodness of segmentation depends on the local probability distributions of the granulometricmoment images. Both exact and asymptotic characterizations of these distributions are developed for the mean image of a basic convexity model.

Head movement estimation for wearable eye tracker

In the study of eye movements in natural tasks, where subjects are able to freely move in their environment, it is desirable to capture a video of the surroundings of the subject not limited to a small field of view as obtained by the scene camera of an eye tracker. Moreover, recovering the head movements could give additional information about the type of eye movement that was carried out, the overall gaze change in world coordinates, and insight into high-order perceptual strategies. Algorithms for the classification of eye movements in such natural tasks could also benefit form the additional head movement data.We propose to use an omnidirectional vision sensor consisting of a small CCD video camera and a hyperbolic mirror. The camera is mounted on an ASL eye tracker and records an image sequence at 60 Hz. Several algorithms for the extraction of rotational motion from this image sequence were implemented and compared in their performance against the measurements of a Fasttrack magnetic tracking system. Using data from the eye tracker together with the data obtained by the omnidirectional image sensor, a new algorithm for the classification of different types of eye movements based on a Hidden-Markov-Model was developed.

Fixation-identification in dynamic scenes: comparing an automated algorithm to manual coding

Video-based eye trackers produce an output video showing where a subject is looking, the subjects point-of-regard (POR), for each frame of a video of the scene. Fixation-identification algorithms simplify the long list of POR data into a more manageable set of data, especially for further analysis, by grouping PORs into fixations. Most current fixation-identification algorithms assume that the POR data are defined in static two-dimensional scene images and only use these raw POR data to identify fixations. The applicability of these algorithms to gaze data in dynamic scene videos is largely unexplored. We implemented a simple velocity-based, duration-sensitive fixation-identification algorithm and compared its performance to results obtained by three experienced users manually coding the eye tracking data displayed within the scene video such that these manual coders had knowledge of the scene motion. We performed this comparison for eye tracking data collected during two different tasks involving different types of scene motion. These two tasks included a subject walking around a building for about 100 seconds (Task 1) and a seated subject viewing a computer animation (approximately 90 seconds long, Task 2). It took our manual coders on average 75 minutes (stdev = 28) and 80 minutes (17) to code results from the first and second tasks, respectively. The automatic fixation-identification algorithm, implemented in MATLAB and run on an Apple 2.16 GHz MacBook, produced results in 0.26 seconds for Task 1 and 0.21 seconds for Task 2. For the first task (walking), the average percent difference among the three human manual coders was 9% (3.5) and the average percent difference between the automatically generated results and the three coders was 11% (2.0). For the second task (animation), the average percent difference among the three human coders was 4% (0.75) and the average percent difference between the automatically generated results and the three coders was 5% (0.9).

Using human observer eye movements in automatic image classifiers

We explore the way in which people look at images of different semantic categories and directly relate those results to computational approaches for automatic image classification. Our hypothesis is that the eye movements of human observers differ for images of different semantic categories, and that this information can be effectively used in automatic content-based classifiers. First, we present eye tracking experiments that show the variation in eye movements across different individuals for image of 5 different categories: handshakes, crowd, landscapes, main object in uncluttered background, and miscellaneous. The eye tracking results suggest that similar viewing patterns occur when different subjects view different images in the same semantic category. Using these results, we examine how empirical data obtained from eye tracking experiments across different semantic categories can be integrated with existing computational frameworks, or used to construct new ones. In particular, we examine the Visual Apprentice, a system in which images classifiers are learned form user input as the user defines a multiple level object definition hierarchy based on an object and its parts and labels examples for specific classes. The resulting classifiers are applied to automatically classify new images. Although many eye tracking experiments have been performed, to our knowledge, this is the first study that specifically compares eye movements across categories, and that links category-specific eye tracking results to automatic image classification techniques.