Yanlin Guo

Aerial video surveillance and exploitation

There is growing interest in performing aerial surveillance using video cameras. Compared to traditional framing cameras, video cameras provide the capability to observe ongoing activity within a scene and to automatically control the camera to track the activity. However, the high data rates and relatively small field of view of video cameras present new technical challenges that must be overcome before such cameras can be widely used. In this paper, we present a framework and details of the key components for real-time, automatic exploitation of aerial video for surveillance applications. The framework involves separating an aerial video into the natural components corresponding to the scene. Three major components of the scene are the static background geometry, moving objects, and appearance of the static and dynamic components of the scene. In order to delineate videos into these scene components, we have developed real time, image-processing techniques for 2-D/3-D frame-to-frame alignment, change detection, camera control, and tracking of independently moving objects in cluttered scenes. The geo-location of video and tracked objects is estimated by registration of the video to controlled reference imagery, elevation maps, and site models. Finally static, dynamic and reprojected mosaics may be constructed for compression, enhanced visualization, and mapping applications.

Iris recognition at a distance

We describe experiments demonstrating the feasibility of human iris recognition at up to 10 m distance between subject and camera. The iris images of 250 subjects were captured with a telescope and infrared camera, while varying distance, capture angle, environmental lighting, and eyewear. Automatic iris localization and registration algorithms, in conjunction with a local correlation based matcher, were used to obtain a similarity score between gallery and probe images. Both the area under the receiver operating characteristic (ROC) curve and the Fisher Linear Discriminant were used to measure the distance between authentic and imposter distributions. Among variables studied, database wide experiments reveal no performance degradation with distance, and minor performance degradation with, in order of increasing effect, time (one month), capture angle, and eyewear.

Hybrid stereo camera: an IBR approach for synthesis of very high resolution stereoscopic image sequences

This paper introduces a novel application of IBR technology for efficient rendering of high quality CG and live action stereoscopic sequences. Traditionally, IBR has been applied to render novel views using image and depth based representations of the plenoptic functions. In this work, we present a restricted form of IBR in which lower resolution images for the views to be generated at a very high resolution are assumed to be available. Specifically, the paper addresses the problem of synthesizing stereo IMAX(R)1 3D motion picture images at a standard resolution of 4-6K. At such high resolutions, producing CG content is extremely time consuming and capturing live action requires bulky cameras. We propose a Hybrid Stereo Camera concept in which one view is rendered at the target high resolution but the other is rendered at a much lower resolution. Methods for synthesizing the second view sequence at the target resolution using image analysis and IBR techniques are the focus of this work. The high quality results from the techniques presented in this paper have been visually evaluated in the IMAX 3D large screen projection environment. The paper also highlights generalizations and extensions of the hybrid stereo camera concept.

Vehicle fingerprinting for reacquisition & tracking in videos

Visual recognition of objects through multiple observations is an important component of object tracking. We address the problem of vehicle matching when multiple observations of a vehicle are separated in time such that frames of observations are not contiguous, thus prohibiting the use of standard frame-to-frame data association. We employ features extracted over a sequence during one time interval as a vehicle fingerprint that is used to compute the likelihood that two or more sequence observations are from the same or different vehicles. The challenges of change in pose, aspect and appearances across two disparate observations are handled by combining feature-based quasi-rigid alignment with flexible matching between two or more sequences. The current work uses the domain of vehicle tracking from aerial platforms where typically both the imaging platform and the vehicles are moving and the number of pixels on the object are limited to fairly low resolutions. Extensive evaluation with respect to ground truth is reported in the paper.

PEET: Prototype Embedding and Embedding Transition for Matching Vehicles over Disparate Viewpoints

This paper presents a novel framework, prototype embedding and embedding transition (PEET), for matching objects, especially vehicles, that undergo drastic pose, appearance, and even modality changes. The problem of matching objects seen under drastic variations is reduced to matching embeddings of object appearances instead of matching the object images directly. An object appearance is first embedded in the space of a representative set of model prototypes (prototype embedding (PE)). Objects captured at disparate temporal and spatial sites are embedded in the space of prototypes that are rendered with the pose of the cameras at the respective sites. Low dimensional embedding vectors are subsequently matched. A significant feature of our approach is that no mapping function is needed to compute the distance between embedding vectors extracted from objects viewed from disparate pose and appearance changes, instead, an embedding transition (ET) scheme is utilized to implicitly realize the complex and non-linear mapping with high accuracy. The heterogeneous nature of matching between high-resolution and low-resolution image objects in PEET is discussed, and an unsupervised learning scheme based on the exploitation of the heterogeneous nature is developed to improve the overall matching performance of mixed resolution objects. The proposed approach has been applied to vehicular object classification and query application, and the extensive experimental results demonstrate the efficacy and versatility of the PEET framework.

Matching vehicles under large pose transformations using approximate 3D models and piecewise MRF model

We propose a robust object recognition method based on approximate 3D models that can effectively match objects under large viewpoint changes and partial occlusion. The specific problem we solve is: given two views of an object, determine if the views are for the same or different object. Our domain of interest is vehicles, but the approach can be generalized to other man-made rigid objects. A key contribution of our approach is the use of approximate models with locally and globally constrained rendering to determine matching objects. We utilize a compact set of 3D models to provide geometry constraints and transfer appearance features for object matching across disparate viewpoints. The closest model from the set, together with its poses with respect to the data, is used to render an object both at pixel (local) level and region/part (global) level. Especially, symmetry and semantic part ownership are used to extrapolate appearance information. A piecewise Markov Random Field (MRF) model is employed to combine observations obtained from local pixel and global region level. Belief Propagation (BP) with reduced memory requirement is employed to solve the MRF model effectively. No training is required, and a realistic object image in a disparate viewpoint can be obtained from as few as just one image. Experimental results on vehicle data from multiple sensor platforms demonstrate the efficacy of our method.

Independent motion detection in 3D scenes

Presents an algorithmic approach to the problem of detecting independently moving objects in 3D scenes that are viewed under camera motion. There are two fundamental constraints that can be exploited for the problem: (i) a two- (or multi-)view camera motion constraint (for instance, the epipolar/trilinear constraint), and (ii) a shape constancy constraint. Previous approaches to the problem either only used partial constraints or relied on dense correspondences or flow. We employ both of these fundamental constraints in an algorithm that does not demand a-priori availability of correspondences or flow. Our approach uses the plane-plus-parallax decomposition to enforce the two constraints. It is also demonstrated, for a class of scenes called sparse 3D scenes, in which genuine parallax and independent motions may be confounded, how the plane-plus-parallax decomposition allows progressive introduction and verification of the fundamental constraints. The results of applying the algorithm to some difficult sparse 3D scenes look promising.

Annotation of video by alignment to reference imagery

Video as an entertainment or information source in consumer, military and broadcast television applications is widespread. Typically however, the video is simply presented to the viewer, with only minimal manipulation. Examples include chroma-keying (often used in news and weather broadcasts) where specific color components are detected and used to control the video source. In the past few years, the advent of digital video and increases in computational power has meant that more complex manipulation can be performed. We present some highlights of our work in annotating video by aligning features extracted from the video to a reference set of features. Video insertion and annotation require manipulation of the video stream to composite synthetic imagery and information with real video imagery. The manipulation may involve only the 2D image space or the 3D scene space. The key problems to be solved are: indexing and matching to determine the location of insertion; stable and jitter-free tracking to compute the time variation of the camera; and seamlessly blended insertion for an authentic viewing experience. We highlight our approach to these problems by showing three example scenarios: 2D synthetic pattern insertion in live video; annotation of aerial imagery through geo-registration with stored reference imagery and annotations; and 3D object insertion in a video for a 3D scene.

Learning-based building outline detection from multiple aerial images

This paper presents a method for detecting building outlines using multiple aerial images. Since data-driven techniques may not be able to account for variability of building geometry and appearances, a key insight explored in this paper is a combination of model-based data driven front end with data driven learning in the back end for increased detection accuracy. The three main components of the detection algorithm are: (i) initialization. Image intensity and depth information are integrally used to efficiently detect buildings, and a robust rectilinear path finding algorithm is adopted to obtain good initial outlines. The initialization process involves the following steps: detecting location of buildings, determining the dominant orientations and knot points in the building outline and using these to fit the initial outline; (ii) learning. A compact set of building features are defined and learned from the well-delineated buildings, and a tree-based classifier is applied to the whole region to detect any missing buildings and obtain their rough outlines; and (iii) verification and refinement. Learned features are used to remove falsely detected buildings, and all outlines are refined by the deformation of rectilinear templates. The experiments, with improved detection rate and precise outlines, demonstrate the applicability of our algorithm.

Robust object matching for persistent tracking with heterogeneous features

Tracking objects over a long period of time in realistic environments remains a challenging problem for ground and aerial video surveillance. Matching objects and verifying their identities across multiple spatial and temporal gaps proves to be an effective way to extend tracking range. When an object track is lost due to occlusion or other reasons, we need to learn the object signature and use it to confirm the objects identity against a set of active objects when it appears again. In order to deal with poor image quality and large variations in aerial video tracking, we present in this paper a unified framework that employs a heterogeneous collection of features such as lines, points and regions for robust vehicle matching under variations in illumination, aspect and camera poses. Our approach fully utilizes the characteristics of vehicular objects that consist of relatively large textureless areas delimited by line like features, and demonstrates the important usage of heterogeneous features for different stages of vehicle matching. Experiments demonstrate the enhancement in performance of vehicle identification across multiple sightings using the heterogeneous feature set.