Anne Guérin-Dugué

Modelling Spatio-Temporal Saliency to Predict Gaze Direction for Short Videos

This paper presents a spatio-temporal saliency model that predicts eye movement during video free viewing. This model is inspired by the biology of the first steps of the human visual system. The model extracts two signals from video stream corresponding to the two main outputs of the retina: parvocellular and magnocellular. Then, both signals are split into elementary feature maps by cortical-like filters. These feature maps are used to form two saliency maps: a static and a dynamic one. These maps are then fused into a spatio-temporal saliency map. The model is evaluated by comparing the salient areas of each frame predicted by the spatio-temporal saliency map to the eye positions of different subjects during a free video viewing experiment with a large database (17000 frames). In parallel, the static and the dynamic pathways are analyzed to understand what is more or less salient and for what type of videos our model is a good or a poor predictor of eye movement.

Comparing models for audiovisual fusion in a noisy-vowel recognition task

Audiovisual speech recognition involves fusion of the audio and video sensors for phonetic identification. There are three basic ways to fuse data streams for taking a decision such as phoneme identification: data-to-decision, decision-to-decision, and data-to-data. This leads to four possible models for audiovisual speech recognition, that is direct identification in the first case, separate identification in the second one, and two variants of the third early integration case, namely dominant recoding or motor recoding. However, no systematic comparison of these models is available in the literature. We propose an implementation of these four models, and submit them to a benchmark test. For this aim, we use a noisy-vowel corpus tested on two recognition paradigms in which the systems are tested at noise levels higher than those used for learning. In one of these paradigms, the signal-to-noise ratio (SNR) value is provided to the recognition systems, in the other it is not. We also introduce a new criterion for evaluating performances, based on transmitted information on individual phonetic features. In light of the compared performances of the four models with the two recognition paradigms, we discuss the advantages and drawbacks of these models, leading to proposals for data representation, fusion architecture, and control of the fusion process through sensor reliability.

A Functional and Statistical Bottom-Up Saliency Model to Reveal the Relative Contributions of Low-Level Visual Guiding Factors

When looking at a scene, we frequently move our eyes to place consecutive interesting regions on the fovea, the retina centre. At each fixation, only this specific foveal region is analysed in detail by the visual system. The visual attention mechanisms control eye movements and depend on two types of factor: bottom-up and top-down factors. Bottom-up factors include different visual features such as colour, luminance, edges, and orientations. In this paper, we evaluate quantitatively the relative contribution of basic low-level features as candidate guiding factors to visual attention and hence to eye movements. We also study how these visual features can be combined in a bottom-up saliency model. Our work consists of three interactive parts: a functional saliency model, a statistical model and eye movement data recorded during free viewing of natural scenes. The functional saliency model, inspired by the primate visual system, decomposes a visual scene into different feature maps. The statistical model indicates which features best explain the recorded eye movements. We show an essential role of high frequency luminance and an important contribution of central fixation bias. The relative contribution of features, calculated by the statistical model, is then used to combine the different feature maps into a saliency map. Finally, the comparison between the saliency model and experimental data confirmed the influence of these contributions.

Curvilinear component analysis for high-dimensional data representation: I. Theoretical aspects and practical use in the presence of noise

Starting from a recall of the theoretical framework, this paper presents the conditions and the strategy of implementation of CCA, a recent algorithm for non-linear mapping. Initially developed in a basic form, for non-linear and high-dimensional data sets, the algorithm is here adapted to the general, and more realistic, case of noisy data. This algorithm, which finds the manifold (in particular, the intrinsic dimension) of the data, has proved to be very efficient in the representation of highly folded data structures. We describe here how it can be tuned to find the average manifold and how robust the convergence is. A companion paper (this issue) presents various applications using this property.

Decision-making in information seeking on texts: an eye-fixation-related potentials investigation

Reading on a web page is known to be not linear and people need to make fast decisions about whether they have to stop or not reading. In such context, reading, and decision-making processes are intertwined and this experiment attempts to separate them through electrophysiological patterns provided by the Eye-Fixation-Related Potentials technique (EFRPs). We conducted an experiment in which EFRPs were recorded while participants read blocks of text that were semantically highly related, moderately related, and unrelated to a given goal. Participants had to decide as fast as possible whether the text was related or not to the semantic goal given at a prior stage. Decision making (stopping information search) may occur when the paragraph is highly related to the goal (positive decision) or when it is unrelated to the goal (negative decision). EFRPs were analyzed on and around typical eye fixations: either on words belonging to the goal (target), subjected to a high rate of positive decisions, or on low frequency unrelated words (incongruent), subjected to a high rate of negative decisions. In both cases, we found EFRPs specific patterns (amplitude peaking between 51 to 120 ms after fixation onset) spreading out on the next words following the goal word and the second fixation after an incongruent word, in parietal and occipital areas. We interpreted these results as delayed late components (P3b and N400), reflecting the decision to stop information searching. Indeed, we show a clear spill-over effect showing that the effect on word N spread out on word N + 1 and N + 2.

Categorisation and Retrieval of Scene Photographs from JPEG Compressed Database

Abstract:Natural image categorisation and retrieval is the main challenge for image indexing. With the increase of available images and video databases, there is a real need to, first, organise the database automatically according to different semantic groups, and secondly, to take into account these large databases where most of the data is stored in a compressed form. The global distribution of orientation features is a very powerful tool to semantically organise the database into groups, such as outdoor urban scenes, indoor scenes, ‘closed’ landscapes (valleys, mountains, forests, etc.) and ‘open’ landscapes (deserts, fields, beaches, etc.). The constraint of a JPEG compressed database is completely integrated with an efficient implementation of an orientation estimator in the DCT (Discrete Cosinus Transform) domain. The proposed estimator is analysed from different points of view (accuracy and discrimination power). The images are then globally characterised by a set of a few parameters (two or three), allowing a fast scenes categorisation and organisation which is very robust to the quantisation effect, up to a quality factor of 10 in the JPEG format.

Models for audiovisual fusion in a noisy-vowel recognition task

This paper presents a study of models for audiovisual (AV) fusion in a noisy-vowel recognition task. We progressively elaborate audiovisual models in order to respect the major principle demonstrated by human subjects in speech perception experiments (the “synergy” principle): audiovisual identification should always be more efficient than auditory-alone or visual-alone identification. We first recall that the efficiency of audiovisual speech recognition systems depends on the level at which they fuse sound and image: four AV architectures are presented, and two are selected for the following of the study. Secondly, we show the importance of providing a contextual input linked to the Signal-to-Noise Ratio (SNR) in the fusion process. Then we propose an original approach using an efficient nonlinear dimension reduction algorithm (“curvilinear components analysis”) in order to increase the performances of the two AV architectures. Furthermore, we show that this approach allows an easy and efficient estimation of the reliability of the audio sensor in relation to SNR, that this estimation can be used to control the AV fusion process, and that it significantly improves the AV performances. Hence, altogether, nonlinear dimension reduction, context estimation and control of the fusion process enable us to respect the “synergy” criterion for the two most used architectures.

Multimodal approach to estimate the ocular movements during EEG recordings: A coupled tensor factorization method.

This paper deals with coupled tensor factorization. A relaxed criterion derived from the advanced coupled matrix-tensor factorization (ACMTF) proposed by Acar et al. is described. The proposed relaxed ACMTF (RACMTF) criterion is based on weaker assumptions that are thus more often satisfied when dealing with actual data. Numerical simulations show the benefit of using jointly two data sets when the underlying factors are highly correlated, especially if one of the modality is less noisy than the other one. The proposed method is finally applied on actual Gaze&EEG data to estimate the ocular artifacts into the EEG recordings.

Adaptive Optimization of Neural Algorithms

Learning neural algorithms are generally very simple, but the convergence is not very fast and robust. In this paper we address the important problem of optimum learning rate adjustement according to an adaptive procedure based on gradient method. The basic idea, very simple, which has already been successfully used in Signal Processing, is extended to 2 neural algorithms : Kohonen self-organizing maps and blind separation of sources (Herault-Jutten algorithm). Although this procedure increases the algorithms complexity, it remains very interesting : -the convergence speed is strongly boosted, -the local nature of learning rule is retained, -the method is applicable to some rule, even if we do not know the cost function (error) which is minimized.

Comparison between adjar and xDawn algorithms to estimate Eye-Fixation Related Potentials distorted by overlapping

Eye-Fixation Related Potentials technique is a joint analysis of both electrical brain activities and ocular movements. It allows to extract neural components synchronized with ocular fixations. However, the extracted brain responses, elicited by adjacent fixations, can be distorted by overlapping processes due to short inter fixations intervals. In this work, the ability of two algorithms, Adjar and xDawn are compared to correct these distortions. The Adjar algorithm is based on assumptions regarding the temporal distributions which become too restrictive for EFRP studies. On the hand, the xDawn algorithm is based on a more general and flexible model that is better adapted for EFRP studies.