Quang Nguyen

Matching pursuit based robust acoustic event classification for surveillance systems

Acoustic event classification using matching pursuit and random forest.Event based feature extraction using atom time and frequency information.Superior in classification of human scream from other classes. Display Omitted The ability to automatically recognize acoustic events in real world conditions is an important application of the surveillance systems. This paper presents an acoustic event classification (AEC) method which uses the Matching Pursuit to extract the important Gabor atoms from input audio signals. Rather than extracting features in short-time frames, we apply the matching pursuit to the whole duration of an acoustic event. Information from atoms, such as time, frequency, and amplitude are used to construct time-frequency features. These features capture both spectral and temporal information of the sound event, which is analogous to the spectrogram representation. Experiments were performed on an 8-class database including human scream and gunshot. Under noisy and mismatched conditions, the proposed classification method achieves F1-score of 0.814, which is superior to state-of-the-art methods.

Selection of the Closest Sound Source for Robot Auditory Attention in Multi-source Scenarios

Robotic auditory attention mainly relies on sound source localization using a microphone array. Typically, the robot detects a sound source whenever it emits, estimates its direction, and then turns to that direction to pay attention. However, in scenarios where multiple sound sources emit simultaneously, the robot may have difficulty with selecting a single target source. This paper proposes a novel robot auditory attention system that is based on source distance perception (e.g., selection of the closest among localized sources). Microphone array consists of head- and base-arrays installed in the robot’s head and base, respectively. The difficulty in the attention among multiple sound sources is solved by estimating a binary mask for each source based on the azimuth localization of the head-array. For each individual source represented by a binary mask, elevations of head- and base-array are estimated and triangulated to obtain distance to the robot. Finally, the closest source is determined and its direction is used for controlling the robot. Experiment results clearly show the benefit of the proposed system, on real indoor recordings of two and three simultaneous sound sources, as well as real-time demonstration at a robot exhibition.

An Environmental Sound Source Classification System Based on Mel-Frequency Cepstral Coefficients and Gaussian Mixture Models

Abstract This paper proposed a study of a sound source classification system that has been developed for detecting and identifying the detected sound events in real environments. The proposed system was based on a pattern recognition approach using Gaussian mixture models and Mel-Frequency Cepstral Coefficients (MFCCs) features. We considered eight types of basic sound sources and an external sound. To make the system robust to various types of sound sources, we designed a tree of reference sound models for classification, in which especially generated total three of GMMs for external sounds according to different characteristics of frequency distributions. The performance of the proposed system, evaluated in terms of percent classification, indicated an averaged accuracy of 91.36% for off-line test. Finally, in on-line test our proposed system also showed a good and stable performance in real environments.

Audio-visual integration for human-robot interaction in multi-person scenarios

This paper presents the integration of audio-visual perception components for human robot interaction in the Robot Operating System (ROS). Visual-based nodes consist of skeleton tracking and gesture recognition using a depth camera, and face recognition using an RGB camera. Auditory perception is based on sound source localization using a microphone array. We present an integration framework of these nodes using a top-down hierarchical messaging protocol. On the top of the integration, a message carries information about the number of persons and their corresponding states (who, what, where), which are updated from many low-level perception nodes. The top message is passed to a planning node to make a reaction of the robot, according to the perception about surrounding people. This paper demonstrates human-robot interaction in multi-persons scenario where robot pays its attention to the speaking or waving hand persons. Moreover, this modularization architecture enables reusing modules for other applications. To validate this approach, two sound source localization algorithms are evaluated in real-time where ground-truth localization is provided by the face recognition module.

Distributed sensor networks for multiple human recognition in indoor environments

In this paper, we propose distributed sensor networks (DSNs) capable of performing reliable recognition targeted at multiple humans in the indoor environments. DSNs are composed with combinations of perception sensor units using a RGB-D sensor and a pan-tilt-zoom camera, and a control board to acquire 3W results of who, where, and what information based on audio-visual perception modules. In addition, fusion methods are utilized to associate with multiple human detection and tracking, face identification, and daily activity recognition. By evaluating the performance of DSNs in a classroom setting, it was confirmed that the proposed system can help to perform the tasks of various purposes.

Robust sound localization for various platform of robots using TDOA map adaptation

In realistic environments, mismatches between the calculated angle-TDOA map with its real exact values are the major reason of performance degradation in sound localization. Usually, those mismatches come from some certain configuration errors or deviations caused by the change of environments. To reduce those mismatches, in this paper we proposed an angle-TDOA map adaptation method which can achieve the robust sound localization in various robot platforms (i.e., various types of microphone array configuration). Especially, the proposed method can be easily applied to the sound localization system by using only several sound sources which generated from some known directions. As a result, the proposed method not only showed a good localization performance, but also saved processing time.

Localization and tracking for simultaneous speakers based on time-frequency method and Probability Hypothesis Density filter

In this paper we present the two steps system of localization and tracking to work in context of simultaneous speakers. The localization algorithm is based on time-frequency method which uses an array of three microphones and it enables to locate multiple sound sources in a single time-frame. Localization results with missing detection and clutter are post-processed by the Probability Hypothesis Density (PHD) filter — based tracking algorithm to estimate the smoothed trajectory of each speaker. The experiments carried out on real data recording show that our method outperforms the multi-target particle filter (MTPF) — based algorithm and is effective in practical application of human-robot interaction.

Audio-visual data fusion for tracking the direction of multiple speakers

This paper presents a multi-speakers tracking algorithm using audio-visual data fusion. The audio information is the direction of speakers and the visual information is the direction of detected faces. These observations are used as inputs of the tracking algorithm, which employed the framework of particle filter. For multi-target tracking, we present a flexible data association and data fusion, which can deal with the appearance or absent of any information during tracking process. The experimental results on data collected from a robot platform in a conventional office room confirm a potential application for human-robot interaction.

Recognition of emergency situations using audio–visual perception sensor network for ambient assistive living

In this paper, we present a perception sensor network (PSN) capable of detecting audio- and visual-based emergency situations such as students’ quarrel with scream and punch, and of keeping an effective school safety. As a system aspect, PSN is basically composed of ambient type sensor units using a Kinect, a pan-tilt-zoom camera, and a control board to acquire raw audio signals, color and depth images. Audio signals, which are acquired by the Kinect microphone array, are used in recognizing sound classes and localizing that sound source. Vision signals, which are acquired by the Kinect and PTZ camera stream, are used to detect the location of humans, identify their name and recognize their gestures. In the system, fusion methods are utilized to associate with multiple person detection and tracking, face identification, and audio–visual emergency recognition. Two approaches of matching pursuit algorithm and dense trajectories covariance matrix are also applied for reliably recognizing abnormal activities of students. Through this, human-caused emergencies are detected automatically while identifying human data of occurrence place, subject, and emergency type. Our PSN that consists of four units was used to conduct experiments to detect the designated target with abnormal actions in multi-person scenarios. By evaluating the performance of perception capabilities and integrated system, it was confirmed that the proposed system can help to conduct more meaningful information which can be of substantive support to teachers or staff members in school environments.

Detection of audio-based emergency situations using perception sensor network

This paper presents a perception sensor network (PSN) for detect audio-based emergency situations such as human scream. The PSN consists of multiple units, each has a Kinect and a pan-tilt-zoom camera. Audio signals, which are acquired by the Kinect microphone array, are used in sound source classification and sound source localization. In order to work in multi-person scenarios, we propose an audio-visual fusion method to detect a single speaking person among multiple ones. The PSN system was demonstrated in a scenario having four persons, where the system is able to detect and localize the screaming person and send a robot to that location to check his/her condition.