Dan Schonfeld

Object Trajectory-Based Activity Classification and Recognition Using Hidden Markov Models

Motion trajectories provide rich spatiotemporal information about an objects activity. This paper presents novel classification algorithms for recognizing object activity using object motion trajectory. In the proposed classification system, trajectories are segmented at points of change in curvature, and the subtrajectories are represented by their principal component analysis (PCA) coefficients. We first present a framework to robustly estimate the multivariate probability density function based on PCA coefficients of the subtrajectories using Gaussian mixture models (GMMs). We show that GMM-based modeling alone cannot capture the temporal relations and ordering between underlying entities. To address this issue, we use hidden Markov models (HMMs) with a data-driven design in terms of number of states and topology (e.g., left-right versus ergodic). Experiments using a database of over 5700 complex trajectories (obtained from UCI-KDD data archives and Columbia University Multimedia Group) subdivided into 85 different classes demonstrate the superiority of our proposed HMM-based scheme using PCA coefficients of subtrajectories in comparison with other techniques in the literature.

Optimal morphological pattern restoration from noisy binary images

A theoretical analysis of morphological filters for the optimal restoration of noisy binary images is presented. The problem is formulated in a general form, and an optimal solution is obtained by using fundamental tools from mathematical morphology and decision theory. Consideration is given to the set-difference distance function as a measure of comparison between images. This function is used to introduce the mean-difference function as a quantitative measure of the degree of geometrical and topological distortion introduced by morphological filtering. It is proved that the class of alternating sequential filters is a set of parametric, smoothing morphological filters that best preserve the crucial structure of input images in the least-mean-difference sense. >

Statistical sequential analysis for real-time video scene change detection on compressed multimedia bitstream

The increased availability and usage of multimedia information have created a critical need for efficient multimedia processing algorithms. These algorithms must offer capabilities related to browsing, indexing, and retrieval of relevant data. A crucial step in multimedia processing is that of reliable video segmentation into visually coherent video shots through scene change detection. Video segmentation enables subsequent processing operations on video shots, such as video indexing, semantic representation, or tracking of selected video information. Since video sequences generally contain both abrupt and gradual scene changes, video segmentation algorithms must be able to detect a large variety of changes. While existing algorithms perform relatively well for detecting abrupt transitions (video cuts), reliable detection of gradual changes is much more difficult. A novel one-pass, real-time approach to video scene change detection based on statistical sequential analysis and operating on a compressed multimedia bitstream is proposed. Our approach models video sequences as stochastic processes, with scene changes being reflected by changes in the characteristics (parameters) of the process. Statistical sequential analysis is used to provide an unified framework for the detection of both abrupt and gradual scene changes.

Robust Video Stabilization Based on Particle Filter Tracking of Projected Camera Motion

Video stabilization is an important technique in digital cameras. Its impact increases rapidly with the rising popularity of handheld cameras and cameras mounted on moving platforms (e.g., cars). Stabilization of two images can be viewed as an image registration problem. However, to ensure the visual quality of the whole video, video stabilization has a particular emphasis on the accuracy and robustness over long image sequences. In this paper, we propose a novel technique for video stabilization based on the particle filtering framework. We extend the traditional use of particle filters in object tracking to tracking of the projected affine model of the camera motions. We rely on the inverse of the resulting image transform to obtain a stable video sequence. The correspondence between scale-invariant feature transform points is used to obtain a crude estimate of the projected camera motion. We subsequently postprocess the crude estimate with particle filters to obtain a smooth estimate. It is shown both theoretically and experimentally that particle filtering can reduce the error variance compared to estimation without particle filtering. The superior performance of our algorithm over other methods for video stabilization is demonstrated through computer simulated experiments.

Real-Time Motion Trajectory-Based Indexing and Retrieval of Video Sequences

This paper presents a novel motion trajectory-based compact indexing and efficient retrieval mechanism for video sequences. Assuming trajectory information is already available, we represent trajectories as temporal ordering of subtrajectories. This approach solves the problem of trajectory representation when only partial trajectory information is available due to occlusion. It is achieved by a hypothesis testing-based method applied to curvature data computed from trajectories. The subtrajectories are then represented by their principal component analysis (PCA) coefficients for optimally compact representation. Different techniques are integrated to index and retrieve subtrajectories, including PCA, spectral clustering, and string matching. We assume a query by example mechanism where an example trajectory is presented to the system and the search system returns a ranked list of most similar items in the dataset. Experiments based on datasets obtained from University of California at Irvines KDD archives and Columbia Universitys DVMM group demonstrate the superiority of our proposed PCA-based approaches in terms of indexing and retrieval times and precision recall ratios, when compared to other techniques in the literature

Fast object tracking using adaptive block matching

We propose a fast object tracking algorithm that predicts the object contour using motion vector information. The segmentation step common in region-based tracking methods is avoided, except for the initialization of the object. Tracking is achieved by predicting the object boundary using block motion vectors followed by updating the contour using occlusions/disocclusion detection. An adaptive block-based approach has been used for estimating motion between frames. An efficient modulation scheme is used to control the gap between frames used for motion estimation. The algorithm for detecting disocclusion proceeds in two steps. First, uncovered regions are estimated from the displaced frame difference. These uncovered regions are classified into actual disocclusions and false alarms by observing the motion characteristics of uncovered regions. Occlusion and disocclusion are considered as dual events and this relationship is explained in detail. The algorithm for detecting occlusion is developed by modifying the disocclusion detection algorithm in accordance with the duality principle. The overall tracking algorithm is computationally superior to existing region-based methods for object tracking. The immediate applications of the proposed tracking algorithm are video compression using MPEG-4 and content retrieval based on standards like H.264. Preliminary simulation results demonstrate the performance of the proposed algorithm.

Segmented trajectory based indexing and retrieval of video data

In this paper, we present a novel principal component analysis (PCA) based approach towards modeling the object trajectory in a video clip. An eigenspace decomposition of high-dimensional trajectory data leads to very compact representation, which is then used as indexing structure. To cutback on PCA computation during indexing, we first segment the trajectories into atomic subtrajectories using a curvature zero-crossing based approach followed by clustering of these subtrajectories. A two-level PCA operation with coarse-to-fine retrieval for query trajectory is then performed to generate retrieval results. Our experimental results show that our global PCA based approach performs better when input query trajectory is of similar length compared to the matching trajectories in the database. However, when partial trajectories are posed as queries our segmented trajectory based approach provides superior results for all precision-recall ratios.

Morphological representation of discrete and binary images

A general theory for the morphological representation of discrete and binary images is presented. The basis of this theory relies on the generation of a set of nonoverlapping segments of an image via repeated erosions and set transformations, which in turn produces a decomposition that guarantees exact reconstruction. The relationship between the proposed representation and some existing shape analysis tools (e.g., discrete size transform, pattern spectrum, skeletons) is investigated, thus introducing the representation as the basis of a unified theory for geometrical image analysis. Particular cases of the general representation scheme are shown to yield a number of useful image decompositions which are directly related to various forms of morphological skeletons. The relationship between the representation and the various forms of morphological skeletons is studied. As a result of this study, a unified theory is developed for the mathematical description of the morphological skeleton decomposition of discrete and binary images. >

View-invariant motion trajectory-based activity classification and recognition

Motion trajectories provide rich spatio-temporal information about an objects activity. The trajectory information can be obtained using a tracking algorithm on data streams available from a range of devices including motion sensors, video cameras, haptic devices, etc. Developing view-invariant activity recognition algorithms based on this high dimensional cue is an extremely challenging task. This paper presents efficient activity recognition algorithms using novel view-invariant representation of trajectories. Towards this end, we derive two Affine-invariant representations for motion trajectories based on curvature scale space (CSS) and centroid distance function (CDF). The properties of these schemes facilitate the design of efficient recognition algorithms based on hidden Markov models (HMMs). In the CSS-based representation, maxima of curvature zero crossings at increasing levels of smoothness are extracted to mark the location and extent of concavities in the curvature. The sequences of these CSS maxima are then modeled by continuous density (HMMs). For the case of CDF, we first segment the trajectory into subtrajectories using CDF-based representation. These subtrajectories are then represented by their Principal Component Analysis (PCA) coefficients. The sequences of these PCA coefficients from subtrajectories are then modeled by continuous density hidden Markov models (HMMs). Different classes of object motions are modeled by one Continuous HMM per class where state PDFs are represented by GMMs. Experiments using a database of around 1750 complex trajectories (obtained from UCI-KDD data archives) subdivided into five different classes are reported.

HMM-based motion recognition system using segmented PCA

In this paper, we propose a novel technique for model-based recognition of complex object motion trajectories using hidden Markov models (HMM). We build our models on principal component analysis (PCA)-based representation of trajectories after segmenting them into small units of perceptually similar pieces of motions. These subtrajectories are then grouped using spectral clustering to decide on the number of states for each HMM representing a class of object motion. The hidden states of the HMMs are represented by Gaussian mixtures (GMs). This way the HMM topology as well as the parameters are automatically derived from the training data in a fully unsupervised sense. Experiments are performed on two data sets; the ASL data set (from UCIs KDD archives) consists of 207 trajectories depicting signs for three words, from Australian Sign Language (ASL); the HJSL data set contains 108 trajectories from sports videos. Our experiments yield an accuracy of 90+% performing much better than existing approaches.