Victor Budkov

Speaker Turn Detection Based on Multimodal Situation Analysis

The main stage of speaker diarization is a detection of time labels, where speakers are changed. The most of the approaches to the decision of the speaker turn detection problem is focused on processing of audio signal captured in one channel and applied for archive records. Recently the problem of speaker diarization became to be considered from multimodal point of view. In this paper we outline modern methods of audio and video signal processing and personification data analysis for multimodal speaker diarization. The proposed PARAD-R software for Russian speech analysis implemented for audio speaker diarization and will be enhanced based on advances of multimodal situation analysis in a meeting room.

Multichannel system of audio-visual support of remote mobile participant at e-meeting

Web-based collaboration using the wireless devices that have multimedia playback capabilities is a viable alternative to traditional face-to-face meetings. E-meetings are popular in businesses because of their cost savings. To provide quick and effective engagement to the meeting activity, the remote user should be able to perceive whole events in the meeting room and have the same possibilities like participants inside. The technological framework of the developed intelligent meeting room implements multichannel audio-visual system for participant activity detection and automatically composes actual multimedia content for remote mobile user. The developed web-based application for remote user interaction with equipment of the intelligent meeting room and organization of E-meetings were tested with Nokia mobile phones.

Event-driven content management system for smart meeting room

Context awareness is one of the key issues at the development of a content management system for meeting recording and teleconference support. An analysis of participant behavior including position in the meeting room, speech activity, face direction, usage of a projector or whiteboard allows the content management system to select actual multimedia streams for recording. The system operation diagram explaining meaningful events for context prediction and equipment actions during the meeting are considered. Experimental results have shown that the graphical content was selected correctly in 97% of whole meeting time.

Context-Aware Mobile Applications for Communication in Intelligent Environment

Peculiarities of communication between users and devices in intelligent environment are described in the paper. The main problems are the heterogeneity of hardware and software at integration of mobile devices as well as the variability of natural signals at the development of multimodal user interfaces. The modern context-aware applications take into account preferences and abilities of user and adapt ones to conditions of physical environments, computing and network resources. The means of Android-based mobile devices, which could be used to acquire attributes of context, are analyzed.

Using a High-Speed Video Camera for Robust Audio-Visual Speech Recognition in Acoustically Noisy Conditions

The purpose of this study is to develop a robust audio-visual speech recognition system and to investigate the influence of a high-speed video data on the recognition accuracy of continuous Russian speech under different noisy conditions. Developed experimental setup and collected multimodal database allow us to explore the impact brought by the high-speed video recordings with various frames per second (fps) starting from standard 25 fps up to high-speed 200 fps. At the moment there is no research objectively reflecting the dependence of the speech recognition accuracy from the video frame rate. Also there are no relevant audio-visual databases for model training. In this paper, we try to fill in this gap for continuous Russian speech. Our evaluation experiments show the increase of absolute recognition accuracy up to 3% and prove that the use of the high-speed camera JAI Pulnix with 200 fps allows achieving better recognition results under different acoustically noisy conditions.

Method of Defining Multimodal Information Falsity for Smart Telecommunication Systems

In the article, a review of the existing methods of transmitted information falsity diagnostics is presented. A conclusion concerning the purposefulness of this function realization in polymodal infocommunication systems has been drawn. A method of defining the multimodal information falsity transmitted in the course of communication act with the help of these systems has been suggested. Common tendencies concerning the dynamics of subscribers’ non-verbal behavior parameters have been formulated. Based on the factor and multiple regressive analysis, the factors depending on such dynamics have been distinguished. Based on the carried out research, a conclusion concerning the possibility of realization of transmitted information falsity in the course of interpersonal communication between subscribers has been drawn and a decisive rule has been formulated.

PARAD-R: Speech analysis software for meeting support

The main goal of the development of systems of formal logging activities is to automate the whole process of transcription of the participant speech. In this paper we outline modern methods of audio and video signal processing and personification data analysis for multimodal speaker diarization. The proposed PARAD-R software for Russian speech analysis implemented for audio speaker diarization and will be enhanced based on advances of multimodal situation analysis in a meeting room.

Mechanical Leg Design of the Anthropomorphic Robot Antares

An overview of the existing anthropomorphic robots and an analysis of servomechanisms and bearing parts involved in the assembly of robot legs are presented. We propose an option for constructing the legs of the robot Antares under development. A two-motor layout, used in the knee, ensures higher joint power along with independent interaction with the neighboring upper and lower leg joints when bending. To reduce the electrical load on the main battery of the robot, the upper legs are provided with a mounting pad for additional batteries powering servos. Direct control of the servos is also carried out through the sub-controllers, responsible for all 6 engines installed in the articular joints of the robot legs.

Methodology of Facility Automation Based on Audiovisual Analysis and Space-Time Structuring of Situation in Meeting Room

Space-time context structurization is one of the key issues of the development of an automatic audiovisual monitoring system for meeting supporting and analysis of participants’ behavior in a smart space. An analysis of accumulated data about participant’s behavior including position in the meeting room, speech activity, and face direction allows monitoring system to generate participant profile, which is further used for predicting his/her behavior on successive meetings. It is also used for an adjustment of an audio and video recording and of multimedia devices controlling model in the smart room. In the experiment the main attention was paid to speaker localization in the chair zone with 32 predefined positions.

Audiovisual speaker localization in medium smart meeting room

The issue of automatic selection of the current active speaker among more than thirty participants located in the medium-sized meeting room is considered. Techniques of video tracking and sound source localization are implemented for recording AVI files of speaker remarks in the developed smart meeting room. Video processing of streams from five cameras serves for registration of participants in fixed chair positions, tracking main speaker based on histogram comparison and AdaBoosted cascade classifier for face detection. Multichannel sound source localization based on GCC-PHAT method is used for estimation of the speaker position by four microphone arrays. In the 18dB SNR case the sound source localization rate was about 97% and fine RMSE was lower 0.23 m.