Claudio M. Privitera

Algorithms for defining visual regions-of-interest: comparison with eye fixations

Many machine vision applications, such as compression, pictorial database querying, and image understanding, often need to analyze in detail only a representative subset of the image, which may be arranged into sequences of loci called regions-of-interest (ROIs). We have investigated and developed a methodology that serves to automatically identify such a subset of aROIs (algorithmically detected ROIs) using different image processing algorithms (IPAs), and appropriate clustering procedures. In human perception, an internal representation directs top-down, context-dependent sequences of eye movements to fixate on similar sequences of hROIs (human identified ROIs). In the paper, we introduce our methodology and we compare aROIs with hROIs as a criterion for evaluating and selecting bottom-up, context-free algorithms. An application is finally discussed.

Pupil dilation during visual target detection

It has long been documented that emotional and sensory events elicit a pupillary dilation. Is the pupil response a reliable marker of a visual detection event while viewing complex imagery? In two experiments where viewers were asked to report the presence of a visual target during rapid serial visual presentation (RSVP), pupil dilation was significantly associated with target detection. The amplitude of the dilation depended on the frequency of targets and the time of target presentation relative to the start of the trial. Larger dilations were associated with trials having fewer targets and with targets viewed earlier in the run. We found that dilation was influenced by, but not dependent on, the requirement of a button press. Interestingly, we also found that dilation occurred when viewers fixated a target but did not report seeing it. We will briefly discuss the role of noradrenaline in mediating these pupil behaviors.

Representation of human vision in the brain: How does human perception recognize images?

The repetitive scanpath eye movement, EM, sequence enabled an approach to the representation of visual images in the human brain. We supposed that there were several levels of binding—semantic or symbolic binding; structural binding for the spatial locations of the regions-of-interest; and sequential binding for the dynamic execution program that yields the sequence of EMs. The scanpath sequences enable experimental evaluation of these various bindings that appear to play independent roles and are likely located in different parts of the modular cortex. EMs play an essen- tial role in top-down control of the flow of visual information. The scanpath theory proposes that an internal spatial-cognitive model controls perception and the active looking EMs. Evidence support- ing the scanpath theory includes experiments with ambiguous fig- ures, visual imagery, and dynamic scenes. It is further explicated in a top-down computer vision tracking scheme for telerobots using design elements from the scanpath procedures. We also introduce procedures—calibration of EMs, identification of regions-of-interest, and analysis and comparison programs—for studying scanpaths. Although philosophers have long speculated that we see in our minds eye, yet until the scanpath theory, no strong scientific evi- dence was available to support these conjectures.

Evaluating image processing algorithms that predict regions of interest

Abstract Several bottom-up, context-free, algorithms for the detection of regions of interest in pictures were analyzed, evaluated and compared. Our aim is to develop new criteria related to human performance for these algorithms and perhaps to be able to design more biologically plausible perceptive machines. We introduce the statistical and computational platform we have been using to compare sequences of regions of interest, both biological (eye movements) and artificial (algorithms).

Neural saccadic response estimation during natural viewing.

Studying neural activity during natural viewing conditions is not often attempted. Isolating the neural response of a single saccade is necessary to study neural activity during natural viewing; however, the close temporal spacing of saccades that occurs during natural viewing makes it difficult to determine the response to a single saccade. Herein, a general linear model (GLM) approach is applied to estimate the EEG neural saccadic response for different segments of the saccadic main sequence separately. It is determined that, in visual search conditions, neural responses estimated by conventional event-related averaging are significantly and systematically distorted relative to GLM estimates due to the close temporal spacing of saccades during visual search. Before the GLM is applied, analyses are applied that demonstrate that saccades during visual search with intersaccadic spacings as low as 100-150 ms do not exhibit significant refractory effects. Therefore, saccades displaying different intersaccadic spacings during visual search can be modeled using the same regressor in a GLM. With the use of the GLM approach, neural responses were separately estimated for five different ranges of saccade amplitudes during visual search. Occipital responses time locked to the onsets of saccades during visual search were found to account for, on average, 79 percent of the variance of EEG activity in a window 90-200 ms after the onsets of saccades for all five saccade amplitude ranges that spanned a range of 0.2-6.0 degrees. A GLM approach was also used to examine the lateralized ocular artifacts associated with saccades. Possible extensions of the methods presented here to account for the superposition of microsaccades in event-related EEG studies conducted in nominal fixation conditions are discussed.

The fixation and saccade P3.

Although most instances of object recognition during natural viewing occur in the presence of saccades, the neural correlates of objection recognition have almost exclusively been examined during fixation. Recent studies have indicated that there are post-saccadic modulations of neural activity immediately following eye movement landing; however, whether post-saccadic modulations affect relatively late occurring cognitive components such as the P3 has not been explored. The P3 as conventionally measured at fixation is commonly used in brain computer interfaces, hence characterizing the post-saccadic P3 could aid in the development of improved brain computer interfaces that allow for eye movements. In this study, the P3 observed after saccadic landing was compared to the P3 measured at fixation. No significant differences in P3 start time, temporal persistence, or amplitude were found between fixation and saccade trials. Importantly, sensory neural responses canceled in the target minus distracter comparisons used to identify the P3. Our results indicate that relatively late occurring cognitive neural components such as the P3 are likely less sensitive to post saccadic modulations than sensory neural components and other neural activity occurring shortly after eye movement landing. Furthermore, due to the similarity of the fixation and saccade P3, we conclude that the P3 following saccadic landing could possibly be used as a viable signal in brain computer interfaces allowing for eye movements.

Decision-Level Fusion of EEG and Pupil Features for Single-Trial Visual Detection Analysis

Several recent studies have reported success in applying EEG-based signal analysis to achieve accurate single-trial classification of responses to visual target detection. Pupil responses are proposed as a complementary modality that can support improved accuracy of single-trial signal analysis. We develop a pupillary response feature-extraction and -selection procedure that helps to improve the classification performance of a system based only on EEG signal analysis. We apply a two-level linear classifier to obtain cognitive-task-related analysis of EEG and pupil responses. The classification results based on the two modalities are then fused at the decision level. Here, the goal is to support increased classification confidence through the inherent modality complementarities. The fusion results show significant improvement over classification performance based on a single modality.

The pupil dilation response to visual detection

The pupil dilation reflex is mediated by inhibition of the parasympathetic Edinger-Westphal oculomotor complex and sympathetic activity. It has long been documented that emotional and sensory events elicit a pupillary reflex dilation. Is the pupil response a reliable marker of a visual detection event? In two experiments where viewers were asked to report the presence of a visual target during rapid serial visual presentation (RSVP), pupil dilation was significantly associated with target detection. The amplitude of the dilation depended on the frequency of targets and the time of the detection. Larger dilations were associated with trials having fewer targets and with targets viewed earlier during the trial. We also found that dilation was strongly influenced by the visual task.

Human-vision-based selection of image processing algorithms for planetary exploration

Independent and automatic image processing is a fundamental objective of the computer vision community. Understanding the role of the eye movement scan path in human vision is an important step toward the achievement of this objective. This top-down model of higher human vision is a new approach to bottom-up image processing algorithms and provides an important new metric and tool in computer vision. We have demonstrated that a small and manageable collection of image processing algorithms, experimentally selected and then combined together can serve in a task such as predicting human eye fixations identifying geological features. Thus, automatic picture analysis based upon human vision could be an essential element in planetary exploration.

Focused JPEG encoding based upon automatic preidentified regions of interest

We have developed a focused-procedure based upon a collection of image processing algorithms that serve to identify regions-of-interest (ROIs), over a digital image. To loci of these ROIs are quantitatively compared with ROIs identified by human eye fixations or glimpses while subjects were looking at the same digital images. The focused- procedure is applied to adjust and adapt the compression ratio over a digital image: - high resolution and poor compression for ROIs; low resolution and strong compression for the major expanse of the entire image. In this way, an overall high compression ratio can be achieved, while at the same time preserving, important visual information within particularly relevant regions of the image. We have bundled the focused-procedures with JPEG, so that the JPEG version allows the result of the compression to be formatted into a file compatible for standard JPEG decoding. Thus, once the image has been compressed, it can be read without difficulty.