Rachid Benmokhtar

Evidence Theory-Based Multimodal Emotion Recognition

Automatic recognition of human affective states is still a largely unexplored and challenging topic. Even more issues arise when dealing with variable quality of the inputs or aiming for real-time, unconstrained, and person independent scenarios. In this paper, we explore audio-visual multimodal emotion recognition. We present SAMMI, a framework designed to extract real-time emotion appraisals from non-prototypical, person independent, facial expressions and vocal prosody. Different probabilistic method for fusion are compared and evaluated with a novel fusion technique called NNET. Results shows that NNET can improve the recognition score (CR + ) of about 19% and the mean average precision of about 30% with respect to the best unimodal system.

Neural network combining classifier based on dempster-shafer theory for semantic indexing in video content

Classification is a major task in many applications and in particular for automatic semantic-based video content indexing and retrieval. In this paper, we focus on the challenging task of classifier output fusion. It is a necessary step to efficiently estimate the semantic content of video shots from multiple cues. We propose to fuse the numeric information provided by multiple classifiers in the framework of evidence logic. For this purpose, an improved version of RBF network based on Evidence Theory (NN-ET) is proposed. Experiments are conducted in the framework of TrecVid high level feature extraction task that consists of ordering shots with respect to their relevance to a given semantic class.

An ontology-based evidential framework for video indexing using high-level multimodal fusion

This paper deals with information retrieval and semantic indexing of multimedia documents. We propose a generic scheme combining an ontology-based evidential framework and high-level multimodal fusion, aimed at recognising semantic concepts in videos. This work is represented on two stages: First, the adaptation of evidence theory to neural network, thus giving Neural Network based on Evidence Theory (NNET). This theory presents two important information for decision-making compared to the probabilistic methods: belief degree and system ignorance. The NNET is then improved further by incorporating the relationship between descriptors and concepts, modeled by a weight vector based on entropy and perplexity. The combination of this vector with the classifiers outputs, gives us a new model called Perplexity-based Evidential Neural Network (PENN). Secondly, an ontology-based concept is introduced via the influence representation of the relations between concepts and the ontological readjustment of the confidence values. To represent this relationship, three types of information are computed: low-level visual descriptors, concept co-occurrence and semantic similarities. The final system is called Ontological-PENN. A comparison between the main similarity construction methodologies are proposed. Experimental results using the TRECVid dataset are presented to support the effectiveness of our scheme.

Classifier fusion: combination methods for semantic indexing in video content

Classifier combination has been investigated as a new research field to improve recognition reliability by taking into account the complementarity between classifiers, in particular for automatic semantic-based video content indexing and retrieval. Many combination schemes have been proposed in the literature according to the type of information provided by each classifier as well as their training and adaptation abilities. This paper presents an overview of current research in classifier combination and a comparative study of a number of combination methods. A novel training technique called Weighted Ten Folding based on Ten Folding principle is proposed for combining classifier. Experiments are conducted in the framework of the TRECVID 2005 features extraction task that consists in ordering shots with respect to their relevance to a given class. Finally, we show the efficiency of different combination methods.

Integrating Image Segmentation and Classification for Fuzzy Knowledge-Based Multimedia Indexing

In this paper we propose a methodology for semantic indexing of images, based on techniques of image segmentation, classification and fuzzy reasoning. The proposed knowledge-assisted analysis architecture integrates algorithms applied on three overlapping levels of semantic information: i) no semantics, i.e. segmentation based on low-level features such as color and shape, ii) mid-level semantics, such as concurrent image segmentation and object detection, region-based classification and, iii) rich semantics, i.e. fuzzy reasoning for extraction of implicit knowledge. In that way, we extract semantic description of raw multimedia content and use it for indexing and retrieval purposes, backed up by a fuzzy knowledge repository. We conducted several experiments to evaluate each technique, as well as the whole methodology in overall and, results show the potential of our approach.

Low-level feature fusion models for soccer scene classification

This paper presents an automatic semantic concept extraction method which employs low level visual feature fusion. Both static and dynamic feature fusion approaches are studied and evaluated. The main contributions of this paper are: a novel dynamic feature fusion approach inspired from coding is proposed to create compact yet rich signatures; Statistical study of descriptors with and without fusion. To validate and evaluate our approach, we have conducted a set experiments on the classification of soccer video shots. These experiments show, in particular, that the feature fusion step of our system increases the classification rate of 17% comparing to a system without feature fusion.

Hierarchical Ontology-Based Robust Video Shots Indexation Using Global MPEG-7 Visual Descriptors

This paper proposes to improve our previous work on the concept-based video shot indexing, by considering an ontological concept construction in the TRECVid 2007 video retrieval, based on two steps. First, each single concept is modeled independently. Second, an ontology-based concept is introduced via the representation of the influence relations between concepts and the ontological readjustment of the confidence values. The main contribution of this paper is in the exploitation manner of the inter-concepts similarity in our indexing system, where three measures are represented: co-occurrence, visual similarity and LSCOM-lite ontology path length contribution. The experimental results report the efficiency and the significant improvement provided by the proposed scheme.

Multi-level Fusion for Semantic Video Content Indexing and Retrieval

In this paper, we present the results of our work on the analysis of an automatic semantic video content indexing and retrieval system based on fusing various low level visual descriptors. Global MPEG-7 features extracted from video shots, are described via IVSM signature (Image Vector Space Model) in order to have a compact description of the content. Both static and dynamic feature fusion are introduced to obtain effective signatures. Support Vector Machines (SVMs) are employed to perform classification (One classifier per feature). The task of the classifiers is to detect the video semantic content. Then, classifier outputs are fused using a neural network based on evidence theory (NNET) in order to provide a decision on the content of each shot. The experimental results are conducted in the framework of the TRECVid feature extraction task.

Video shots key-frames indexing and retrieval through pattern analysis and fusion techniques

This paper proposes an automatic semantic video content indexing and retrieval system based on fusing various low level visual and shape descriptors. Extracted features from region and sub-image blocks segmentation of video shots key-frames are described via IVSM signature (Image Vector Space Model) in order to have a compact and efficient description of the content. Static feature fusion based on averaging and concatenation are introduced to obtain effective signatures. Support Vector Machines (SVM) and neurals network (NNs) are employed to perform classification. The task of the classifiers is to detect the video semantic content. Then, classifiers outputs are fused using neural network based on evidence theory (NN-ET) in order to provide a decision on the content of each shot. The experimental results are conducted in the framework of soccer video feature extraction task.

Ontological reranking approach for hybrid concept similarity-based video shots indexation

This paper proposes to compare three hybrid concept similarity measures for video shots indexing and retrieval [1], based on two steps. First, individuals concepts are modeled independently. Second, an ontology is introduced via the representation of the relationship between concepts and the ontological readjustment of the confidence values. Our contribution lies in the manner in which inter-concepts similarities are exploited in the indexing system using co-occurrence, visual descriptors, and hybrid semantic similarities. Experimental results report the efficiency and the significant improvement provided by the proposed scheme.