Alexandrina Rogozan

Adaptive fusion of acoustic and visual sources for automatic speech recognition

Among the various methods proposed to improve the accuracy and the robustness of automatic speech recognition (ASR), the use of additional knowledge sources is a successful one. In particular, a recent method proposes supplementing the acoustic information with visual data mostly derived from the speakers lip shape. Perceptual studies support this approach by emphasising the importance of visual information for speech recognition in humans. This paper describes a method we have developed for adaptive integration of acoustic and visual information in ASR. Each modality is involved in the recognition process with a different weight, which is dynamically adapted during this process mainly according to the signal-to-noise ratio provided as a contextual input. We tested this method on continuous hidden Markov model-based systems developed according to direct identification (DI), separate identification (SI) and hybrid identification (DI + SI) strategies. Experiments performed under various noise-level conditions show that the DI + SI based system is the most promising one when compared to both DI and SI-based systems for a speaker-dependent continuous-spelling of French letters recognition task. They also confirm that using adaptive modality weights instead of fixed weights allows for performance improvement and that weight estimation could benefit from using visemes as decision units for the visual recogniser in SI and DI + SI based systems.

Sensors for Obstacle Detection - A Survey

The obstacle detection field is a very broad one and a lot of obstacle detection systems have been developed in the last years in this domain. We tried to identify the main character of an obstacle detection system from the ruttier scene. Thus, we classified the main types of sensors from this field in passive (visible and infrared spectrum camera) and active (radar, laser-scanner, sonar) sensors and we made a survey in this domain. After a short presentation of every type of sensor, we presented another current and fancy solution for an obstacle detection system: the fusion of different sensor together. Almost all obstacle detection systems use a combination of passive-active technology, and in general the best solution is obtained using a vision system combined with a distance sensor like radar or laser. Maybe the most low-priced system is one combining only vision systems, but the inconvenient in this case is the lack of distance information.

DISCRIMINATIVE LEARNING OF VISUAL DATA FOR AUDIOVISUAL SPEECH RECOGNITION

In recent years a number of techniques have been proposed to improve the accuracy and the robustness of automatic speech recognition in noisy environments. Among these, suplementing the acoustic information with visual data, mostly extracted from speakers lip shapes, has been proved to be successful. We have already demonstrated the effectiveness of integrating visual data at two different levels during speech decoding according to both direct and separate identification strategies (DI+SI). This paper outlines methods for reinforcing the visible speech recognition in the framework of separate identification. First, we define visual-specific units using a self-organizing mapping technique. Second, we complete a stochastic learning of these units with a discriminative neural-network-based technique for speech recognition purposes. Finally, we show on a connected-letter speech recognition task that using these methods improves performances of the DI+SI based system under varying noise-level conditions.

Evolving kernel functions for SVMs by genetic programming

hybrid model for evolving support vector machine (SVM) kernel functions is developed in this paper. The kernel expression is considered as a parameter of the SVM algorithm and the current approach tries to find the best expression for this SVM parameter. The model is a hybrid technique that combines a genetic programming (GP) algorithm and a support vector machine (SVM) algorithm. Each GP chromosome is a tree encoding the mathematical expression for the kernel function. The evolved kernel is compared to several human-designed kernels and to a previous genetic kernel on several datasets. Numerical experiments show that the SVM embedding our evolved kernel performs statistically better than standard kernels, but also than previous genetic kernel for all considered classification problems.

Combining SURF-based local and global features for road obstacle recognition in far infrared images

This paper describes a road obstacle classification system that recognizes both vehicles and pedestrians in far-infrared images. Different local and global features based on Speeded Up Robust Features (SURF) were investigated and then selected in order to extract a discriminative signature from the infrared spectrum. First, local features representing the local appearance of an obstacle, are extracted from a codebook of scale and rotation-invariant SURF features. Second, global features were used since they provide complementary information by characterizing shape and texture. When compared with the state-of-the-art Haar and Gabor wavelet features, our method provides significant improvement of recognition performances. Moreover, since our SURF based representation is invariant to the scale and the number of local features extracted from objects, our system performs the recognition task without resizing images. Our system was evaluated on a set of far-infrared images where obstacles occur at different scales and in difficult recognition situations. By using a multi-class SVM approach, accuracy rates of 91.51% has been achieved on Surf-based representation, while a maximum rate of 89.11% was achieved on wavelet-based representation.

A robust cost function for stereo matching of road scenes

In this paper different matching cost functions used for stereo matching are evaluated in the context of intelligent vehicles applications. Classical costs are considered, like: sum of squared differences, normalised cross correlation or Census Transform that were already evaluated in previous studies, together with some recent functions that try to enhance the discriminative power of Census Transform (CT). These are evaluated with two different stereo matching algorithms: a global method based on graph cuts and a fast local one based on cross aggregation regions. Furthermore we propose a new cost function that combines the CT and alternatively a variant of CT called Cross-Comparison Census (CCC), with the mean sum of relative pixel intensity differences (DIFFCensus). Among all the tested cost functions, under the same constraints, the proposed DIFFCensus produces the lower error rate on the KITTI road scenes dataset with both global and local stereo matching algorithms.

Improving SVM Performance Using a Linear Combination of Kernels

Standard kernel-based classifiers use only a single kernel, but the real-world applications and the recent developments of various kernel methods have emphasized the need to consider a combination of multiple kernels. We propose an evolutionary approach for finding the optimal weights of a combined kernel used by the Support Vector Machines (SVM) algorithm for solving some particular problems. We use a genetic algorithm (GA) for evolving these weights. The numerical experiments show that the evolved combined kernels (ECKs) perform better than the convex combined kernels (CCKs) for several classification problems.

Intensity self similarity features for pedestrian detection in Far-Infrared images

Pedestrian detection is an important but challenging component of an Intelligent Transportation System. In this paper, we describe a pedestrian detection system based on a monocular vision with a Far-Infrared camera (FIR). We propose an original feature representation, called Intensity Self Similarity (ISS), adapted to pedestrian detection in FIR images. The ISS representation is based on the relative intensity self similarity within a pedestrian region of interest (ROI) hypothesis. Our system consists of two components. The first component generates pedestrian ROI hypothesis by exploiting the specific characteristics of FIR images, where pedestrian shapes may vary in large scale, but heads appear usually as light regions. Pedestrian ROI are detected, with high recall rate, due to a Hierarchical Codebook (HC) of Speeded-Up Robust Features (SURF) located in light head regions. The second component consists of pedestrian hypothesis validation, by using a pedestrian full-body classification based on the ISS representation, with Support Vector Machine (SVM). For classification, we retained two feature descriptors: the Histogram of Oriented Gradients (HOG) descriptor and the original ISS feature representation that we proposed for FIR images. The early fusion of these two features enhances significantly the system precision, attaining an F-measure for the pedestrian class of 97.7%. Moreover, this feature fusion outperforms the state-of-the-art SURF descriptor proposed previously. The experimental evaluation shows that our pedestrian detector is also robust, since it performs well in detecting pedestrians even in large scale and crowded real-world scenes.

Pedestrian recognition based on hierarchical codebook of SURF features in visible and infrared images

One of the main challenges in Intelligent Vehicle is recognition of road obstacles. Our goal is to design a real-time, precise and robust pedestrian recognition system. We choose to use Speeded Up Robust Features (SURF) and a Support Vector Machine (SVM) classifier in order to perform the recognition task. Our main contribution is a method for fast computation of discriminative features for pedestrian recognition. Fast features extraction is assured by using a hierarchical codebook of scale and rotation-invariant SURF features. We evaluate our approach for pedestrian recognition in a set of images where people occur at different scales and in difficult recognition situations. The system shows good performance in visible and especially in infrared images. Besides, experimental results show that the hierarchical structure presents a major interest not only for maintaining a reasonable feature extraction time, but also for improving classification results.

Asynchronous integration of visual information in an automatic speech recognition system

Deals with that integration of visual data in automatic speech recognition systems. We first describe the framework of our research; the development of advanced multi-user multi-modal interfaces. Then we present audio-visual speech recognition problems in general, and the ones we are interested in, in particular. After a very brief discussion of existing systems, we present the architecture of our audio-only reference and baseline systems and describe our audio-visual systems. The major part of the paper describes the systems we developed according to two different approaches to the problem of integration of visual data in speech recognition systems. We first describe a system we developed according to the first approach (called the direct integration model) and show its limitations. Our approach, which we call asynchronous integration, is then presented. After the general guidelines, we go into some details about the distributed architecture and the variant of the N-best algorithm we developed for the implementation of this approach. The performances of these different systems are compared, and we conclude by a brief discussion of the performance improvements we have obtained and future work.