A. G. Amitha Perera

A large-scale benchmark dataset for event recognition in surveillance video

We introduce a new large-scale video dataset designed to assess the performance of diverse visual event recognition algorithms with a focus on continuous visual event recognition (CVER) in outdoor areas with wide coverage. Previous datasets for action recognition are unrealistic for real-world surveillance because they consist of short clips showing one action by one individual [15, 8]. Datasets have been developed for movies [11] and sports [12], but, these actions and scene conditions do not apply effectively to surveillance videos. Our dataset consists of many outdoor scenes with actions occurring naturally by non-actors in continuously captured videos of the real world. The dataset includes large numbers of instances for 23 event types distributed throughout 29 hours of video. This data is accompanied by detailed annotations which include both moving object tracks and event examples, which will provide solid basis for large-scale evaluation. Additionally, we propose different types of evaluation modes for visual recognition tasks and evaluation metrics along with our preliminary experimental results. We believe that this dataset will stimulate diverse aspects of computer vision research and help us to advance the CVER tasks in the years ahead.

Multi-Object Tracking Through Simultaneous Long Occlusions and Split-Merge Conditions

A fundamental requirement for effective automated analysis of object behavior and interactions in video is that each object must be consistently identified over time. This is difficult when the objects are often occluded for long periods: nearly all tracking algorithms will terminate a track with loss of identity on a long gap. The problem is further confounded by objects in close proximity, tracking failures due to shadows, etc. Recently, some work has been done to address these issues using higher level reasoning, by linking tracks from multiple objects over long gaps. However, these efforts have assumed a one-to-one correspondence between tracks on either side of the gap. This is often not true in real scenarios of interest, where the objects are closely spaced and dynamically occlude each other, causing trackers to merge objects into single tracks. In this paper, we show how to efficiently handle splitting and merging during track linking. Moreover, we show that we can maintain the identities of objects that merge together and subsequently split. This enables the identity of objects to be maintained throughout long sequences with difficult conditions. We demonstrate our approach on a highly challenging, oblique-view video sequence of dense traffic of a highway interchange. We successfully track the large majority of the hundreds of moving vehicles in the scene, many in close proximity, through long occlusions and shadows.

A unified framework for tracking through occlusions and across sensor gaps

A common difficulty encountered in tracking applications is how to track an object that becomes totally occluded, possibly for a significant period of time. Another problem is how to associate objects, or tracklets, across non-overlapping cameras, or between observations of a moving sensor that switches fields of regard. A third problem is how to update appearance models for tracked objects over time. As opposed to using a comprehensive multi-object tracker that must simultaneously deal with these tracking challenges, we present a novel, modular framework that handles each of these problems in a unified manner by the initialization, tracking, and linking of high-confidence tracklets. In this track/suspend/match paradigm, we first analyze the scene to identify areas where tracked objects are likely to become occluded. Tracking is then suspended on occluded objects and re-initiated when they emerge from behind the occlusion. We then associate, or match, suspended tracklets with the new tracklets using full kinematic models for object motion and Gibbsian distributions for object appearance in order to complete the track through the occlusion. Sensor gaps are handled in a similar manner, where tracking is suspended when the sensor looks away and then re-initiated when the sensor returns. Changes in object appearance and orientation during tracking are also seamlessly handled in this framework. Tracklets with low lock scores are terminated. Tracking then resumes on untracked movers with corresponding updated appearance models. These new tracklets are then linked back to the terminated ones as appropriate. Fully automatic tracking results from a moving sensor are presented.

Detecting and counting people in surveillance applications

A number of surveillance scenarios require the detection and tracking of people. Although person detection and counting systems are commercially available today, there is need for further research to address the challenges of real world scenarios. The focus of this work is the segmentation of groups of people into individuals. One relevant application of this algorithm is people counting. Experiments document that the presented approach leads to robust people counts.

Stand-off Iris Recognition System

The iris is a highly accurate biometric identifier. However widespread adoption is hindered by the difficulty of capturing high-quality iris images with minimal user co-operation. This paper describes a first-generation prototype iris identification system designed for stand-off cooperative access control. This system identifies individuals who stand in front of and face the system after 3.2 seconds on average. Subjects within a capture zone are imaged with a calibrated pair of wide-field-of-view surveillance cameras. A subject is located in three dimensions using face detection and triangulation. A zoomed near infrared iris camera on a pan-tilt platform is then targeted to the subject. The iris camera lens has its focal distance automatically adjusted based on the subject distance. Integrated with the iris camera on the pan-tilt platform is a near infrared illuminator that is composed of an array of directed LEDs. Video frames from the iris camera are processed to detect and segment the iris, generate a template and then identify the subject.

Multimedia event detection with multimodal feature fusion and temporal concept localization

We present a system for multimedia event detection. The developed system characterizes complex multimedia events based on a large array of multimodal features, and classifies unseen videos by effectively fusing diverse responses. We present three major technical innovations. First, we explore novel visual and audio features across multiple semantic granularities, including building, often in an unsupervised manner, mid-level and high-level features upon low-level features to enable semantic understanding. Second, we show a novel Latent SVM model which learns and localizes discriminative high-level concepts in cluttered video sequences. In addition to improving detection accuracy beyond existing approaches, it enables a unique summary for every retrieval by its use of high-level concepts and temporal evidence localization. The resulting summary provides some transparency into why the system classified the video as it did. Finally, we present novel fusion learning algorithms and our methodology to improve fusion learning under limited training data condition. Thorough evaluation on a large TRECVID MED 2011 dataset showcases the benefits of the presented system.

Learning Motion Patterns in Surveillance Video using HMM Clustering

We present a novel approach to learning motion behavior in video, and detecting abnormal behavior, using hierarchical clustering of hidden Markov models (HMMs). A continuous stream of track data is used for online and on-demand creation and training of HMMs, where tracks may be of highly variable length and scenes may be very complex with an unknown number of motion patterns. We show how these HMMs can be used for on-line clustering of tracks that represent normal behavior and for detection of deviant tracks. The track clustering algorithm uses a hierarchical agglomerative HMM clustering technique that jointly determines all the HMM parameters (including the number of states) via an expectation maximization (EM) algorithm and the Akaike information criteria. Results are demonstrated on a highly complex scene containing dozens of routes, significant occlusions and hundreds of moving objects.

An intelligent video framework for homeland protection

This paper presents an overview of Intelligent Video work currently under development at the GE Global Research Center and other research institutes. The image formation process is discussed in terms of illumination, methods for automatic camera calibration and lessons learned from machine vision. A variety of approaches for person detection are presented. Crowd segmentation methods enabling the tracking of individuals through dense environments such as retail and mass transit sites are discussed. It is shown how signature generation based on gross appearance can be used to reacquire targets as they leave and enter disjoint fields of view. Camera calibration information is used to further constrain the detection of people and to synthesize a top-view, which fuses all camera views into a composite representation. It is shown how site-wide tracking can be performed in this unified framework. Human faces are an important feature as both a biometric identifier and as a method for determining the focus of attention via head pose estimation. It is shown how automatic pan-tilt- zoom control; active shape/appearance models and super-resolution methods can be used to enhance the face capture and analysis problem. A discussion of additional features that can be used for inferring intent is given. These include body-part motion cues and physiological phenomena such as thermal images of the face.

Joint Recognition of Complex Events and Track Matching

We present a novel method for jointly performing recognition of complex events and linking fragmented tracks into coherent, long-duration tracks. Many event recognition methods require highly accurate tracking, and may fail when tracks corresponding to event actors are fragmented or partially missing. However, these conditions occur frequently from occlusions, traffic and tracking errors. Recently, methods have been proposed for linking track fragments from multiple objects under these difficult conditions. Here, we develop a method for solving these two problems jointly. A hypothesized event model, represented as a Dynamic Bayes Net, supplies data-driven constraints on the likelihood of proposed track fragment matches. These event-guided constraints are combined with appearance and kinematic constraints used in the previous track linking formulation. The result is the most likely track linking solution given the event model, and the highest event score given all of the track fragments. The event model with the highest score is determined to have occurred, if the score exceeds a threshold. Results demonstrated on a busy scene of airplane servicing activities, where many non-event movers and long fragmented tracks are present, show the promise of the approach to solving the joint problem.

Micro-calcification detection in digital tomosynthesis mammography

A novel technique for the detection and enhancement of microcalcifications in digital tomosynthesis mammography (DTM) is presented. In this method, the DTM projection images are used directly, instead of using a 3D reconstruction. Calcification residual images are computed for each of the projection images. Calcification detection is then performed over 3D space, based on the values of the calcification residual images at projection points for each 3D point under test. The quantum, electronic, and tissue noise variance at each pixel in each of the calcification residuals is incorporated into the detection algorithm. The 3D calcification detection algorithm finds a minimum variance estimate of calcification attenuation present in 3D space based on the signal and variance of the calcification residual images at the corresponding points in the projection images. The method effectively detects calcifications in 3D in a way that both ameliorates the difficulties of joint tissue/microcalcification tomosynthetic reconstruction (streak artifacts, etc.) and exploits the well understood image properties of microcalcifications as they appear in 2D mammograms. In this method, 3D reconstruction and calcification detection and enhancement are effectively combined to create a calcification detection specific reconstruction. Motivation and details of the technique and statistical results for DTM data are provided.