Thomas V. Papathomas

The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments

This paper presents the theory, design principles, implementation and performance results of PicHunter, a prototype content-based image retrieval (CBIR) system. In addition, this document presents the rationale, design and results of psychophysical experiments that were conducted to address some key issues that arose during PicHunters development. The PicHunter project makes four primary contributions to research on CBIR. First, PicHunter represents a simple instance of a general Bayesian framework which we describe for using relevance feedback to direct a search. With an explicit model of what users would do, given the target image they want, PicHunter uses Bayess rule to predict the target they want, given their actions. This is done via a probability distribution over possible image targets, rather than by refining a query. Second, an entropy-minimizing display algorithm is described that attempts to maximize the information obtained from a user at each iteration of the search. Third, PicHunter makes use of hidden annotation rather than a possibly inaccurate/inconsistent annotation structure that the user must learn and make queries in. Finally, PicHunter introduces two experimental paradigms to quantitatively evaluate the performance of the system, and psychophysical experiments are presented that support the theoretical claims.

Implicit attentional selection of bound visual features.

Traditionally, research on visual attention has been focused on the processes involved in conscious, explicit selection of task-relevant sensory input. Recently, however, it has been shown that attending to a specific feature of an object automatically increases neural sensitivity to this feature throughout the visual field. Here we show that directing attention to a specific color of an object results in attentional modulation of the processing of task-irrelevant and not consciously perceived motion signals that are spatiotemporally associated with this color throughout the visual field. Such implicit cross-feature spreading of attention takes place according to the veridical physical associations between the color and motion signals, even under special circumstances when they are perceptually misbound. These results imply that the units of implicit attentional selection are spatiotemporally colocalized feature clusters that are automatically bound throughout the visual field.

Motion processing by chromatic and achromatic visual pathways.

We describe a family of stimuli consisting of colored bars of different orientations, which, when presented in rapid succession, may elicit unambiguous motion perception. These stimuli permitted the isolation of directional spatiotemporal information extracted from oriented luminance clues, from nonoriented chromatic-plus-luminance clues, or, when the stimuli were presented under equiluminant conditions, from pure chromatic clues. As a general rule, matching of orientation induces weaker motion-detection performances than does matching of color. When the orientation clues are in competition with the chromatic ones, motion perception based on the former is always overridden by motion perception based on the latter. We indirectly isolated an oriented chromatic mechanism that also contributes to motion perception. We finally showed that, under equiluminant conditions, matching of orientation across different colors is inefficient in eliciting motion perception, either because motion information is extracted poorly across different chromatic channels or because such channels show little orientational selectivity. Because motion strength determined by each of the manipulated attributes follows different functions with the displacement (or velocity) of the stimuli, we propose the existence of three underlying mechanisms, a luminance mechanism, a chromatic-plus-luminance mechanism, and a pure chromatic mechanism, each of which provides motion information.

Two carriers for motion perception: Color and luminance

Starting with the experiments of Ramachandran and Gregory (Nature, 275, 55-56, 1978), several psychophysical studies in apparent motion (AM) have established that the perception of motion is significantly impaired at equiluminance. Still debated, however, is whether color alone can resolve ambiguities in AM. We report here on several psychophysical experiments, the quantitative results of which indicate that color does play a substantial role in AM. These findings seem to support recently proposed neurophysiological frameworks according to which there exist significant interactions among the neuronal pathways mediating the perception of basic visual attributes such as color, motion, form and depth.

Reduced Depth Inversion Illusions in Schizophrenia Are State-Specific and Occur for Multiple Object Types and Viewing Conditions

Schizophrenia patients are less susceptible to depth inversion illusions (DIIs) in which concave faces appear as convex, but what stimulus attributes generate this effect and how does it vary with clinical state? To address these issues, we had 30 schizophrenia patients and 25 well-matched healthy controls make convexity judgments on physically concave faces and scenes. Patients were selectively sampled from three levels of care to ensure symptom heterogeneity. Half of the concave objects were painted with realistic texture to enhance the convexity illusion; the remaining objects were painted uniform beige to reduce the illusion. Subjects viewed the objects with one eye while laterally moving in front of the stimulus (to see depth via motion parallax) or with two eyes while remaining motionless (to see depth stereoscopically). For each group, DIIs were stronger with texture than without, and weaker with stereoscopic information than without, indicating that patients responded normally to stimulus alterations. More importantly, patients experienced fewer illusions than controls irrespective of the face/scene category, texture, or viewing condition (parallax/stereo). Illusions became less frequent as patients experienced more positive symptoms and required more structured treatment. Taken together, these results indicate that people with schizophrenia experience fewer DIIs with a variety of object types and viewing conditions, perhaps because of a lessened tendency to construe any type of object as convex. Moreover, positive symptoms and the need for structured treatment are associated with more accurate 3-D perception, suggesting that DII may serve as a state marker for the illness.

Hidden annotation in content based image retrieval

The Bayesian relevance feedback approach introduced with the PicHunter system (Cox, Miller, Omohundro and Yianilos, Int. Conf. on Pattern Recognition, pp. 361-369, 1996) is extended to include hidden semantic attributes. The general approach is motivated and experimental results are presented that demonstrate significant reductions in search times (28-32%) using these annotations

Experiments with a Hollow Mask and a Reverspective: Top-down Influences in the Inversion Effect for 3-D Stimuli

Earlier psychophysical and physiological studies, obtained mostly with two-dimensional (2-D) stimuli, provided evidence for the hypothesis that the processing of faces differs from that of scenes. We report on our experiments, employing realistic three-dimensional (3-D) stimuli of a hollow mask and a scene, that offer further evidence for this hypothesis. The stimuli used for both faces and scenes were bistable, namely they could elicit either the veridical or an illusory volumetric percept. Our results indicate that the illusion is weakened when the stimuli are inverted, suggesting the involvement of top-down processes. This inversion effect is statistically significant for the facial stimulus, but the trend did not reach statistical significance for the scene stimulus. These results support the hypothesis that configural processing is stronger for the 3-D perception of faces than it is for scenes, and extend the conclusions of earlier studies on 2-D stimuli.

Experiments on the role of painted cues in Hughes's reverspectives

The English artist Patrick Hughes has created an extraordinary class of painted artpieces, most commonly referred to as ‘reverspectives’. They consist of truncated pyramids and prisms with their smaller faces closer to the viewer, in such a way as to allow a realistic scene to be painted on them. The works of art contain rich perspective and other painted cues that conspire to elicit an illusory depth percept that is the reverse of the physical depth arrangement. This reverse depth is obtained under a wide range of viewing conditions, and competes with the veridical depth percept in a classical bistable paradigm that was found to exhibit a high degree of hysteresis. Under the illusory depth percept, reverspectives appear to move vividly as the viewer moves in front of them. This paper reports two experiments that were designed to assess the effectiveness of the painted cues in eliciting the illusory depth percept by using three different measures for the strength of the illusion. As expected, the illusion was favored by monocular viewing and large viewing distances. The results from these two experiments are in close agreement with each other, and they indicate that the painted cues are powerful in influencing the ultimate percept.

Object-based cross-feature attentional modulation from color to motion.

Object-based theories of visual attention predict that attempting to direct attention to a particular attribute of a visual object will result in an automatic selection of the whole object, including all of its features. It has been assumed, but not critically tested, that the spreading of attention from one feature to another in this manner, i.e. cross-feature attentional (CFA) effects, takes place at object-level stages of processing as opposed to early, local stages. In the present study we disambiguated these options for color-to-motion CFA by contrasting attentions effect on bivectorial transparent versus bivectorial locally paired motion displays. We found that association between features at the global, but not at the local, stage of motion processing leads to cross-feature attentional effects. These findings provide strong psychophysical evidence that such effects are indeed object-based.

Audiovisual short-term influences and aftereffects in motion: Examination across three sets of directional pairings

The study of cross-modal influences in perception, particularly between the auditory and visual modalities, has been intensified recently. This paper reports on a comprehensive study of auditory-visual cross-modal influences in motion, including motion aftereffects (MAE). We examined both auditory influences on visual perception and vice versa. Visual motion interactions were examined using three directional pairings or configurations: along the horizontal, vertical, and depth axes. In Experiment 1 we assessed how the simultaneous presence of a strong motion signal in one modality affected the perception of motion in the other modality. To investigate further whether such influences have long-term effects, we tested whether adaptation in one modality alone could produce cross-modal MAEs in Experiment 2. Overall, the pattern of results was similar across all directional pairings, with the strongest cross-modal influences observed in motion along the horizontal axis; this is likely due to the greater co-localization of the two stimuli in this configuration. Although both auditory and visual stimuli affected the other modality when presented simultaneously, significant cross-modally induced aftereffects could only be produced using visual stimuli. However, we did observe vertical visual MAE following adaptation to auditory spectral motion. These results are discussed in terms of current psychophysical and neurophysiological findings concerning the way in which auditory-visual signals are processed.