Jwu-Sheng Hu

Robust background subtraction with shadow and highlight removal for indoor surveillance

This work describes a robust background subtraction scheme involving shadow and highlight removal for indoor environmental surveillance. Foreground regions can be precisely extracted by the proposed scheme despite illumination variations and dynamic background. The Gaussian mixture model (GMM) is applied to construct a color-based probabilistic background model (CBM). Based on CBM, the short-term color-based background model (STCBM) and the long-term color-based background model (LTCBM) can be extracted and applied to build the gradient-based version of the probabilistic background model (GBM). Furthermore, a new dynamic cone-shape boundary in the RGB color space, called a cone-shape illumination model (CSIM), is proposed to distinguish pixels among shadow, highlight, and foreground. A novel scheme combining the CBM, GBM, and CSIM is proposed to determine the background which can be used to detect abnormal conditions. The effectiveness of the proposed method is demonstrated via experiments with several video clips collected in a complex indoor environment.

Simultaneous localization of mobile robot and multiple sound sources using microphone array

Sound source localization is an important function in robot audition. The existing works perform sound source localization using static microphone arrays. This work proposes a framework that simultaneously localizes the mobile robot and multiple sound sources using a microphone array on the robot. First, an eigenstructure-based generalized cross correlation method for estimating time delays between microphones under multi-source environments is described. A method to compute the far field source directions as well as the speed of sound using the estimated time delays is proposed. In addition, the correctness of the sound speed estimate is utilized to eliminate spurious sources, which greatly enhances the robustness of sound source detection. The arrival angles of the detected sound sources are used as observations in a bearings-only SLAM procedure. As the source signals are not persistent and there is no identification of the signal content, data association is unknown which is solved using FastSLAM. The experimental results demonstrate the effectiveness of the proposed approaches.

Simultaneous Localization of a Mobile Robot and Multiple Sound Sources Using a Microphone Array

Sound source localization is an important function in robot audition. The existing works perform sound source localization using static microphone arrays. This work proposes a framework that simultaneously localizes the mobile robot and multiple sound sources using a microphone array on the robot. First, an eigenstructure-based generalized cross correlation method for estimating time delays between microphones under multi-source environments is described. A method to compute the far field source directions as well as the speed of sound using the estimated time delays is proposed. In addition, the correctness of the sound speed estimate is utilized to eliminate spurious sources, which greatly enhances the robustness of sound source detection. The arrival angles of the detected sound sources are used as observations in a bearings-only SLAM procedure. As the source signals are not persistent and there is no identification of the signal content, data association is unknown which is solved using FastSLAM. The experimental results demonstrate the effectiveness of the proposed approaches.

Robust speaker's location detection in a vehicle environment using GMM models

Human-computer interaction (HCI) using speech communication is becoming increasingly important, especially in driving where safety is the primary concern. Knowing the speakers location (i.e., speaker localization) not only improves the enhancement results of a corrupted signal, but also provides assistance to speaker identification. Since conventional speech localization algorithms suffer from the uncertainties of environmental complexity and noise, as well as from the microphone mismatch problem, they are frequently not robust in practice. Without a high reliability, the acceptance of speech-based HCI would never be realized. This work presents a novel speakers location detection method and demonstrates high accuracy within a vehicle cabinet using a single linear microphone array. The proposed approach utilize Gaussian mixture models (GMM) to model the distributions of the phase differences among the microphones caused by the complex characteristic of room acoustic and microphone mismatch. The model can be applied both in near-field and far-field situations in a noisy environment. The individual Gaussian component of a GMM represents some general location-dependent but content and speaker-independent phase difference distributions. Moreover, the scheme performs well not only in nonline-of-sight cases, but also when the speakers are aligned toward the microphone array but at difference distances from it. This strong performance can be achieved by exploiting the fact that the phase difference distributions at different locations are distinguishable in the environment of a car. The experimental results also show that the proposed method outperforms the conventional multiple signal classification method (MUSIC) technique at various SNRs.

A sliding-window visual-IMU odometer based on tri-focal tensor geometry

This paper presents an odometer architecture which combines a monocular camera and an inertial measurement unit (IMU). The trifocal tensor geometry relationship between three images is used as camera measurement information, which makes the proposed method without estimating the 3D position of feature point. In other words, the proposed method does not have to reconstruct environment. Meanwhile, the camera pose corresponding to each of the three images are refined in filter to form a multi-state constraint Kalman filter (MSCKF). Consequently, this paper proposes a sliding window odometry which has a balance between computational cost and accuracy. Compared with traditional visual odometry or simultaneous localization and mapping (SLAM) method, the proposed method not only meets the requirement of odometer in the ego-motion estimation, but also suit for real-time application. This paper further proposes a random sample consensus (RANSAC) algorithm which is based on three views geometry. The RANSAC algorithm can effectively reject feature points which are mismatch or located on independently moving objects, thus it make the overall algorithm capable of operating in dynamic environment. Experiments are conducted to show the effectiveness of the proposed method in real environment.

Speaker attention system for mobile robots using microphone array and face tracking

This paper presents a real-time human-robot interface system (HRIS), which processes both speech and vision information to improve the quality of communication between human and an autonomous mobile robot. The HRIS contains a real-time speech attention system and a real-time face tracking system. In the speech attention system, a microphone-array voice acquisition system has been developed to estimate the direction of speaker and purify the speakers speech signal in a noisy environment. The developed face tracking system aims to track the speakers face under illumination variation and react to the face motion. The proposed HRIS can provide a robot with the abilities of finding a speakers direction, tracking the speakers face, moving its body to the speaker, focusing its attention to the speaker who is talking to it, and purifying the speakers speech. The experimental results show that the HRIS not only purifies speech signal with a significant performance, but also tracks a face under illumination variation in real-time

Ensuring safety in human-robot coexistence environment

This paper proposes a safety index and an associated formulation in the optimization-based path planning framework to assess and ensure the safety of human workers in a human-robot coexistence environment. The safety index is evaluated using the ellipsoid coordinates (EC) attached to the robot links that represents the distance between the robot arm and the worker. To account for the inertial effect, the momentum of the robot links are projected onto the coordinates to generate additional measures of safety. The safety index is used as a constraint in the optimization problem so that a collision-free trajectory within a finite time horizon is generated online iteratively for the robot to move towards the desired position. To reduce the computational load for real-time implementation, the formulated optimization problem is further approximated by a quadratic problem. The safety index and the proposed formulations are simulated and validated in a two-link planar robot and the ITRI 7-DoF robot with a human worker moving inside the workspace of the robots.

Estimation of sound source number and directions under a multisource reverberant environment

Sound source localization is an important feature in robot audition. This work proposes a sound source number and directions estimation method under a multisource reverberant environment. An eigenstructure-based generalized cross-correlation method is proposed to estimate time delay among microphones. A source is considered as a candidate if the corresponding time delay combination among microphones gives reasonable sound speed estimation. Under reverberation, some candidates might be spurious but their direction estimations are not consistent for consecutive data frames. Therefore, an adaptive K-means++ algorithm is proposed to cluster the accumulated results from the sound speed selection mechanism. Experimental results demonstrate the performance of the proposed algorithm in a real room.

A new spatial-color mean-shift object tracking algorithm with scale and orientation estimation

In this paper, we propose a new mean-shift tracking algorithm based on a novel similarity measure function. The joint spatial-color feature is used as our basic model elements. The target image is modeled with the kernel density estimation and the new similarity measure functions is developed using the expectation of the estimated kernel density. With these new similarity measure functions, two similarity-based mean-shift tracking algorithms are derived. To enhance the robustness, the weighted background information is added into the proposed tracking algorithm. In order to solve the object deformation problem, the principal component analysis is used to update the orientation of the tracking object, and corresponding eigenvalues are used to monitor the scale of the object. The experimental results show that the new similarity-based tracking algorithms can be implemented in real-time and are able to track the moving object with an automatic update of the orientation and scale.

Shape Memorization and Recognition of 3D Objects Using a Similarity-Based Aspect-Graph Approach

This paper presents an integrated framework for recognizing 3D objects from 2D images. A flexible combinational algorithm motivated by the novel view expressed by Cyr and Kimia is proposed to generate the aspects of a 3D object as the object prototype using features extracted from the collected 2D images sampled at random intervals from the viewing sphere. Fourier descriptors of the sampled points on the object contour and point-to-point lengths are calculated as the features and similarity metrics are applied to extract the characteristic views as the aspects. Moreover, the object prototype can be integrated from new collected 2D views. Besides, foreground detection with shadow and highlight removal is used to improve the facility of capturing the explicit object efficiently. The effectiveness of the proposed method is demonstrated by experiments with different rigid objects and human postures.