Jaco Vermaak

Color-Based Probabilistic Tracking

Color-based trackers recently proposed in [3,4,5] have been proved robust and versatile for a modest computational cost. They are especially appealing for tracking tasks where the spatial structure of the tracked objects exhibits such a dramatic variability that trackers based on a space-dependent appearance reference would break down very fast. Trackers in [3,4,5] rely on the deterministic search of a window whose color content matches a reference histogram color model.Relying on the same principle of color histogram distance, but within a probabilistic framework, we introduce a new Monte Carlo tracking technique. The use of a particle filter allows us to better handle color clutter in the background, as well as complete occlusion of the tracked entities over a few frames.This probabilistic approach is very flexible and can be extended in a number of useful ways. In particular, we introduce the following ingredients: multi-part color modeling to capture a rough spatial layout ignored by global histograms, incorporation of a background color model when relevant, and extension to multiple objects.

Data fusion for visual tracking with particles

The effectiveness of probabilistic tracking of objects in image sequences has been revolutionized by the development of particle filtering. Whereas Kalman filters are restricted to Gaussian distributions, particle filters can propagate more general distributions, albeit only approximately. This is of particular benefit in visual tracking because of the inherent ambiguity of the visual world that stems from its richness and complexity. One important advantage of the particle filtering framework is that it allows the information from different measurement sources to be fused in a principled manner. Although this fact has been acknowledged before, it has not been fully exploited within a visual tracking context. Here we introduce generic importance sampling mechanisms for data fusion and discuss them for fusing color with either stereo sound, for teleconferencing, or with motion, for surveillance with a still camera. We show how each of the three cues can be modeled by an appropriate data likelihood function, and how the intermittent cues (sound or motion) are best handled by generating proposal distributions from their likelihood functions. Finally, the effective fusion of the cues by particle filtering is demonstrated on real teleconference and surveillance data.

Monte Carlo filtering for multi target tracking and data association

We present Monte Carlo methods for multi-target tracking and data association. The methods are applicable to general nonlinear and non-Gaussian models for the target dynamics and measurement likelihood. We provide efficient solutions to two very pertinent problems: the data association problem that arises due to unlabelled measurements in the presence of clutter, and the curse of dimensionality that arises due to the increased size of the state-space associated with multiple targets. We develop a number of algorithms to achieve this. The first, which we refer to as the Monte Carlo joint probabilistic data association filter (MC-JPDAF), is a generalisation of the strategy proposed by Schulz et al. (2001) and Schulz et al. (2003). As is the case for the JPDAF, the distributions of interest are the marginal filtering distributions for each of the targets, but these are approximated with particles rather than Gaussians. We also develop two extensions to the standard particle filtering methodology for tracking multiple targets. The first, which we refer to as the sequential sampling particle filter (SSPF), samples the individual targets sequentially by utilising a factorisation of the importance weights. The second, which we refer to as the independent partition particle filter (IPPF), assumes the associations to be independent over the individual targets, leading to an efficient component-wise sampling strategy to construct new particles. We evaluate and compare the proposed methods on a challenging synthetic tracking problem.

Nonlinear filtering for speaker tracking in noisy and reverberant environments

This paper addresses the problem of speaker tracking in a noisy and reverberant environment using time delay of arrival (TDOA) measurements at spatially distributed microphone pairs. The tracking problem is posed within a state-space estimation framework, and models are developed for the speaker motion and the likelihood of the speaker location in the light of the TDOA measurements. The resulting state-space model is nonlinear and nonGaussian, and consequently no closed-form solutions exist for the filtering distributions required to perform tracking. Here sequential Monte Carlo (SMC) methods are applied to approximate the true filtering distribution with a set of samples. The resulting tracking algorithm requires no triangulation, is computationally efficient, and can straightforwardly be extended to track multiple speakers.

Particle methods for Bayesian modeling and enhancement of speech signals

This paper applies time-varying autoregressive (TVAR) models with stochastically evolving parameters to the problem of speech modeling and enhancement. The stochastic evolution models for the TVAR parameters are Markovian diffusion processes. The main aim of the paper is to perform on-line estimation of the clean speech and model parameters and to determine the adequacy of the chosen statistical models. Efficient particle methods are developed to solve the optimal filtering and fixed-lag smoothing problems. The algorithms combine sequential importance sampling (SIS), a selection step and Markov chain Monte Carlo (MCMC) methods. They employ several variance reduction strategies to make the best use of the statistical structure of the model. It is also shown how model adequacy may be determined by combining the particle filter with frequentist methods. The modeling and enhancement performance of the models and estimation algorithms are evaluated in simulation studies on both synthetic and real speech data sets.

Sequential Monte Carlo fusion of sound and vision for speaker tracking

Video telephony could be considerably enhanced by provision of a tracking system that allows freedom of movement to the speaker while maintaining a well-framed image, for transmission over limited bandwidth. Already commercial multi-microphone systems exist which track speaker direction in order to reject background noise. Stereo sound and vision are complementary modalities in that sound is good for initialisation (where vision is expensive) whereas vision is good for localisation (where sound is less precise). Using generative probabilistic models and particle filtering, we show that stereo sound and vision can indeed be fused effectively, to make a system more capable than with either modality on its own.

Towards Improved Observation Models for Visual Tracking: Selective Adaptation

An important issue in tracking is how to incorporate an appropriate degree of adaptivity into the observation model. Without any adaptivity, tracking fails when object properties change, for example when illumination changes affect surface colour. Conversely, if an observation model adapts too readily then, during some transient failure of tracking, it is liable to adapt erroneously to some part of the background. The approach proposed here is to adapt selectively, allowing adaptation only during periods when two particular conditions are met: that the object should be both present and in motion. The proposed mechanism for adaptivity is tested here with a foreground colour and motion model. The experimental setting itself is novel in that it uses combined colour and motion observations from a fixed filter bank, with motion used also for initialisation via a Monte Carlo proposal distribution. Adaptation is performed using a stochastic EM algorithm, during periods that meet the conditions above. Tests verify the value of such adaptivity, in that immunity to distraction from clutter of similar colour to the object is considerably enhanced.

Variational inference for visual tracking

The likelihood models used in probabilistic visual tracking applications are often complex non-linear and/or non-Gaussian functions, leading to analytically intractable inference. Solutions then require numerical approximation techniques, of which the particle filter is a popular choice. Particle filters, however, degrade in performance as the dimensionality of the state space increases and the support of the likelihood decreases. As an alternative to particle filters this paper introduces a variational approximation to the tracking recursion. The variational inference is intractable in itself, and is combined with an efficient importance sampling procedure to obtain the required estimates. The algorithm is shown to compare favorably with particle filtering techniques on a synthetic example and two real tracking problems. The first involves the tracking of a designated object in a video sequence based on its color properties, whereas the second involves contour extraction in a single image.

Models and Algorithms for Tracking of Maneuvering Objects Using Variable Rate Particle Filters

Standard algorithms in tracking and other state-space models assume identical and synchronous sampling rates for the state and measurement processes. However, real trajectories of objects are typically characterized by prolonged smooth sections, with sharp, but infrequent, changes. Thus, a more parsimonious representation of a target trajectory may be obtained by direct modeling of maneuver times in the state process, independently from the observation times. This is achieved by assuming the state arrival times to follow a random process, typically specified as Markovian, so that state points may be allocated along the trajectory according to the degree of variation observed. The resulting variable dimension state inference problem is solved by developing an efficient variable rate particle filtering algorithm to recursively update the posterior distribution of the state sequence as new data becomes available. The methodology is quite general and can be applied across many models where dynamic model uncertainty occurs on-line. Specific models are proposed for the dynamics of a moving object under internal forcing, expressed in terms of the intrinsic dynamics of the object. The performance of the algorithms with these dynamical models is demonstrated on several challenging maneuvering target tracking problems in clutter.

A hybrid approach for online joint detection and tracking for multiple targets

In this paper, we present a new approach for online joint detection and tracking for multiple targets. We combine a deterministic clustering algorithm for target detection with a sequential Monte Carlo method for multiple target tracking. The proposed approach continuously monitors the appearance and disappearance of a set of regions of interest for target detection within the surveillance region. No computational effort for target tracking is expended unless these regions of interest are persistently detected. In addition, we also integrate a very efficient 2D data assignment algorithm into the sampling method for the data association problem. The proposed approach is applicable to nonlinear and nonGaussian models for the target dynamics and measurement likelihood. Computer simulations demonstrate that the proposed hybrid approach is robust in performing joint detection and tracking for multiple targets even though the environment is hostile in terms of high clutter density and low target detection probability