Piotr Bratoszewski

Visual lip contour detection for the purpose of speech recognition

A method for visual detection of lip contours in frontal recordings of speakers is described and evaluated. The purpose of the method is to facilitate speech recognition with visual features extracted from a mouth region. Different Active Appearance Models are employed for finding lips in video frames and for lip shape and texture statistical description. Search initialization procedure is proposed and error measure values are monitored in order to prevent the matching process from converging to a false local minimum. AAM-based visual features are applied in an experiment devoted to the static recognition of English vowels with SVM. Studies are carried out based on a database of recordings of 5 speakers of different skin colors. Results are thoroughly discussed and illustrated with figures.

An audio-visual corpus for multimodal automatic speech recognition

A review of available audio-visual speech corpora and a description of a new multimodal corpus of English speech recordings is provided. The new corpus containing 31 hours of recordings was created specifically to assist audio-visual speech recognition systems (AVSR) development. The database related to the corpus includes high-resolution, high-framerate stereoscopic video streams from RGB cameras, depth imaging stream utilizing Time-of-Flight camera accompanied by audio recorded using both: a microphone array and a microphone built in a mobile computer. For the purpose of applications related to AVSR systems training, every utterance was manually labeled, resulting in label files added to the corpus repository. Owing to the inclusion of recordings made in noisy conditions the elaborated corpus can also be used for testing robustness of speech recognition systems in the presence of acoustic background noise. The process of building the corpus, including the recording, labeling and post-processing phases is described in the paper. Results achieved with the developed audio-visual automatic speech recognition (ASR) engine trained and tested with the material contained in the corpus are presented and discussed together with comparative test results employing a state-of-the-art/commercial ASR engine. In order to demonstrate the practical use of the corpus it is made available for the public use.

Massive surveillance data processing with supercomputing cluster

We propose a new model of online surveillance data streams processing on a supercomputer.Massive parallelization enables online object detection in high-resolution video streams.Crowding detection in multi-camera image sequences is performed in nearly real-time.Online privacy protection in surveillance systems is possible on cluster architecture. In recent years, increasingly complex algorithms for automated analysis of surveillance data are being developed. The rapid growth in the number of monitoring installations and higher expectations of the quality parameters of the captured data result in an enormous computational cost of analyzing the massive volume of data. In this paper a new model of online processing of surveillance data streams is proposed, which assumes the use of services running within a supercomputer platform. The study presents some of the highly parallelized algorithms for detecting safety-threatening events in high-resolution-video streams, which were developed during the research, and discusses their performance on the supercomputer platform.

Pilot testing of developed multimodal biometric identity verification system

The bank client identity verification system developed in the course of the IDENT project is presented. The total number of five biometric modalities including: dynamic signature proofing, voice recognition, face image verification, face contour extraction and hand blood vessels distribution comparison have been developed and studied. The experimental data were acquired employing multiple biometric sensors installed at engineered biometric terminals. The biometric portraits of 125 subjects were registered and stored in the database during the presented pilot study and then verified experimentally. The analysis of FAR and FRR measures obtained for developed biometric applications was made. Problem-specific survey was done on the basis of questionnaires completed by the subjects in order to assess the look and feel of the developed biometric system as well as to collect opinions concerning its implementation in banking outlets. A discussion concerning the quality of registered signals and results achieved in the pilot study is included.

Face Profile View Retrieval Using Time of Flight Camera Image Analysis

Method for profile view retrieving of the human face is presented. The depth data from the 3D camera is taken as an input. The preprocessing is, besides of standard filtration, extended by the process of filling of the holes which are present in depth data. The keypoints, defined as the nose tip and the chin are detected in user’s face and tracked. The Kalman filtering is applied to smooth the coordinates of those points which can vary with each frame because of the subject’s movement in front of the camera. Knowing the locations of keypoints and having the depth data the contour of the user’s face a profile retrieval is attempted. Further filtering and modifications are introduced to the profile view in order to enhance its representation. Data processing enhancements allow emphasizing minima and maxima in the contour signals leading to discrimination of the face profiles and enable robust facial landmarks tracking.

Enhanced voice user interface employing spatial filtration of signals from acoustic vector sensor

Spatial filtration of sound is introduced to enhance speech recognition accuracy in noisy conditions. An acoustic vector sensor (AVS) is employed. The signals from the AVS probe are processed in order to attenuate the surrounding noise. As a result the signal to noise ratio is increased. An experiment is featured in which speech signals are disturbed by babble noise. The signals before and after spatial filtration are processed by an automatic speech recognition (ASR) engine. It is shown that employing spatial filtration of signals from the AVS probe leads to a significant reduction in word error rate (WER) for a dictionary of 184 words.

The Project IDENT: Multimodal Biometric System for Bank Client Identity Verification

Biometric identity verification methods are implemented inside a real banking environment comprising: dynamic handwritten signature verification, face recognition, bank client voice recognition and hand vein distribution verification. A secure communication system based on an intra-bank client-server architecture was designed for this purpose. Hitherto achieved progress within the project is reported in this paper with a focus on the design and implementation of the developed biometric authentication system. Implemented multimodal biometric client identity verification methods are briefly outlined and results of hitherto obtained biometric sample acquisition and analysis are reported.

Validating data acquired with experimental multimodal biometric system installed in bank branches

An experimental system was engineered and implemented in 100 copies inside a real banking environment comprising: dynamic handwritten signature verification, face recognition, bank client voice recognition and hand vein distribution verification. The main purpose of the presented research was to analyze questionnaire responses reflecting user opinions on: comfort, ergonomics, intuitiveness and other aspects of the biometric enrollment process. The analytical studies and experimental work conducted in the course of this work will lead towards methodologies and solutions of the multimodal biometric technology, which is planned for further development. Before this stage is achieved a study on the data usefulness acquired from a variety of biometric sensors and from survey questionnaires filled in by banking tellers and clients was done. The decision-related sets were approximated by the Rough Set method offering efficient algorithms and tools for finding hidden patterns in data. Prediction of evaluated biometric data quality, based on enrollment samples and on user subjective opinions was made employing the developed method. After an introduction to the principles of applied biometric identity verification methods, the knowledge modelling approach is presented together with achieved results and conclusions.

Visual Detection of People Movement Rules Violation in Crowded Indoor Scenes

The paper presents a camera-independent framework for detecting violations of two typical people movement rules that are in force in many public transit terminals: moving in the wrong direction or across designated lanes. Low-level image processing is based on object detection with Gaussian Mixture Models and employs Kalman filters with conflict resolving extensions for the object tracking. In order to allow an effective event recognition in a crowded environment, the algorithm for event detection is supplemented with the optical-flow based analysis in order to obtain pixel-level velocity characteristics. The proposed solution is evaluated with multi-camera, real-life recordings from an airport terminal. Results are discussed and compared with a traditional approach that does not include optical flow based direction of movement analysis.

Performance Analysis of Developed Multimodal Biometric Identity Verification System

The bank client identity verification system developed in the course of the IDENT project is presented. The total number of five biometric modalities including: dynamic handwritten signature proofing, voice recognition, face image verification, face contour extraction and hand blood vessels distribution comparison have been developed and studied. The experimental data were acquired employing multiple biometric sensors installed at engineered biometric terminals. The biometric portraits of more than 10 000 bank clients were registered and stored in the database during the presented study and then verified experimentally. Problem- specific survey was done on the basis of questionnaires completed by the subjects in order to assess the look and feel of the developed biometric system as well as to collect opinions concerning its implementation in banking outlets. A discussion concerning the quality of registered data and results achieved in the study is included.