Kohsia S. Huang

Activity monitoring and summarization for an intelligent meeting room

Intelligent meeting rooms should support efficient and effective interactions among their occupants. In this paper, we present our efforts toward building intelligent environments using a multimodal sensor network of static cameras, active (pan/tilt/zoom) cameras and microphone arrays. Active cameras are used to capture details associated with interesting events. The goal is not only to make a system that supports multi-person interactions in the environment in real time, but also to have the system remember the past, enabling reviews of past events in an intuitive and efficient manner. In this paper, we present the system specifications and major components, integration framework, active network control procedures and experimental studies involving multi-person interactions in an intelligent meeting room environment.

Robust real-time detection, tracking, and pose estimation of faces in video streams

Robust human face analysis has been recognized as a crucial part in intelligent systems. In this paper, we present the development of a computational framework for robust detection, tracking, and pose estimation of faces captured by video arrays. We discuss the development of a multi-primitive skin-tone and edge-based detection module embedded in a tracking module for efficient and robust face detection and tracking. A continuous density HMM based pose estimation is developed for an accurate estimate of the face orientation motions. Experimental evaluations of these algorithms suggest the validity of the proposed framework and its computational modules.

Driver's view and vehicle surround estimation using omnidirectional video stream

Our research is focused on the development of novel machine vision based telematic systems, which provide non-intrusive probing of the state of the driver and driving conditions. In this paper we present a system which allows simultaneous capture of the drivers head pose, driving view, and surroundings of the vehicle. The integrated machine vision system utilizes a video stream of full 360 degree panoramic field of view. The processing modules include perspective transformation, feature extraction, head detection, head pose estimation, driving view synthesis, and motion segmentation. The paper presents a multi-state statistical decision models with Kalman filtering based tracking for head pose detection and face orientation estimation. The basic feasibility and robustness of the approach is demonstrated with a series of systematic experimental studies.

3D Shape Context Based Gesture Analysis Integrated with Tracking using Omni Video Array

In this paper we introduce a multilayer cylindrical histogram based feature space for representing 3D shape context of human body. Dynamics of gestures are analyzed using discrete hidden Markov models (DHMM) with quantized feature vectors. Extensive experimental trials with multiple subjects and a range of gesture classes are presented. Gesture recognition accuracies of over 85% (for nine gestures, and 9 subjects) and over 95% (for seven gestures) support the basic feasibility and robustness of the approach.

Streaming face recognition using multicamera video arrays

We present face recognition schemes based on video streams: the majority decision rule and HMM maximum likelihood (ML) decision rules. PCA type of subspace feature analysis is first applied to the face images in a video segment of a fixed number of frames. The majority decision rule is then applied to PCA recognition results in the video segment. Discrete HMM (DHMM) is also applied to the single-frame recognition sequences. Continuous density HMM (CDHMM) is applied directly to the sequence of PCA feature vectors for ML decision on the video segment in a delayed decision manner. Experimental results are compared between these three schemes in terms of the number of states and Gaussian mixtures of the HMMs. CDHMM-based decision rule achieved a 99% correct recognition rate in average. A geometric interpretation of ML in the feature subspace well explains the observed performances.

Networked omnivision arrays for intelligent environment

Intelligent environments are systems that are aware of the spatial information and activities within them through sensors and interact with people in a natural and unobtrusive way. An intelligent system using networked omnivision array is proposed based on specified requirements of intelligent environments. It utilizes Omni-Directional Vision Sensor (ODVS) network as the sensory input. ODVS optical modeling is described, which allows panoramic and perspective view generation. A 3D tracker based on the ODVS network is constructed. Using the tracking information, active camera selection and dynamic perspective view generation enable real-time face tracking. Face recognition is also implemented for person identification. Current results of the modules and extensions to the system are also discussed.

Distributed video arrays for tracking, human identification, and activity analysis

In this paper we discuss the intelligent or smart camera based systems for capturing visual contextual information at multiple levels of information abstraction. A distributed video array, with multiple omnidirectional and rectilinear cameras, is used to acquire visual information. System architecture as well as 3D tracking and human identification modules are described. Examples of the use of distributed video arrays in indoor, outdoor, and mobile environments are presented.

Integrated detection, tracking, and recognition of faces with omnivideo array in intelligent environments

We present a multilevel system architecture for intelligent environments equipped with omnivideo arrays. In order to gain unobtrusive human awareness, real-time 3D human tracking as well as robust video-based face detection and tracking and face recognition algorithms are needed. We first propose a multiprimitive face detection and tracking loop to crop face videos as the front end of our face recognition algorithm. Both skin-tone and elliptical detections are used for robust face searching, and view-based face classification is applied to the candidates before updating the Kalman filters for face tracking. For video-based face recognition, we propose three decision rules on the facial video segments. The majority rule and discrete HMM (DHMM) rule accumulate single-frame face recognition results, while continuous density HMM (CDHMM) works directly with the PCA facial features of the video segment for accumulated maximum likelihood (ML) decision. The experiments demonstrate the robustness of the proposed face detection and tracking scheme and the three streaming face recognition schemes with 99% accuracy of the CDHMM rule. We then experiment on the system interactions with single person and group people by the integrated layers of activity awareness. We also discuss the speech-aided incremental learning of new faces.

Distributed omni-video arrays and digital tele-viewer for customized viewing, event detection and notification

Recent innovations in real-time machine vision, distributed computing, software architectures, and encrypted high-speed networking are expanding the available technology for intelligent camera arrays on televiewing and interactive applications. In this paper we describe research aimed at the realization of a powerful televiewing system using arrays of omnidirectional cameras. The video arrays provide an interactive, real-time, multiresolution televiewing interface to multiple people including physical security and emergency response crews simultaneously. Security measures such as authentication and network encryption are implemented on multiple platforms. Fusion of high-resolution rectilinear image over the omnidirectional image is also carried out. Computer vision techniques based on motion analysis are used for detecting interesting events. The televiewing system is verified for the performance as well as physically tested on the Super Bowl 2003 event for crowd detection and traffic monitoring.