Matthew Brand

Discovery and segmentation of activities in video

Hidden Markov models (HMMs) have become the workhorses of the monitoring and event recognition literature because they bring to time-series analysis the utility of density estimation and the convenience of dynamic time warping. Once trained, the internals of these models are considered opaque; there is no effort to interpret the hidden states. We show that by minimizing the entropy of the joint distribution, an HMMs internal state machine can be made to organize observed activity into meaningful states. This has uses in video monitoring and annotation, low bit-rate coding of scene activity, and detection of anomalous behavior. We demonstrate with models of office activity and outdoor traffic, showing how the framework learns principal modes of activity and patterns of activity change. We then show how this framework can be adapted to infer hidden state from extremely ambiguous images, in particular, inferring 3D body orientation and pose from sequences of low-resolution silhouettes.

A direct method for 3D factorization of nonrigid motion observed in 2D

The nonrigid structure-from-motion (NSFM) problem seeks to recover a sequence of 3D shapes, shape articulation parameters, and camera view matrices from 2D correspondence data. Factorization approaches relate the principal subspaces of the data matrix to the desired parameters through a linear corrective transform. Current methods for finding this transform are heuristic or depend on strong assumptions about the data. We show how to solve for this transform by directly minimizing deviation from the required orthogonal structure of the projection/articulation matrix. The solution is exact for noiseless data and an order of magnitude more accurate than state-of-the-art methods for noisy data.

Flexible flow for 3D nonrigid tracking and shape recovery

We introduce linear methods for model-based tracking of nonrigid 3D objects and for acquiring such models from video. 3D motions and flexions are calculated directly from image intensities without information-lossy intermediate results. Measurement uncertainty is quantified and fully propagated through the inverse model to yield posterior mean (PM) and/or mode (MAP) pose estimates. A Bayesian framework manages uncertainty, accommodates priors, and gives confidence measures. We obtain highly accurate and robust closed-form estimators by minimizing information loss from non-reversible (inner-product and least-squares) operations, and, when unavoidable, performing such operations with the appropriate error norm. For model acquisition, we show how to refine a crude or generic model to fit the video subject. We demonstrate with tracking, model refinement, and super-resolution texture lifting from low-quality low-resolution video.

Lifting 3D Manhattan Lines from a Single Image

We propose a novel and an efficient method for reconstructing the 3D arrangement of lines extracted from a single image, using vanishing points, orthogonal structure, and an optimization procedure that considers all plausible connectivity constraints between lines. Line detection identifies a large number of salient lines that intersect or nearly intersect in an image, but relatively a few of these apparent junctions correspond to real intersections in the 3D scene. We use linear programming (LP) to identify a minimal set of least-violated connectivity constraints that are sufficient to unambiguously reconstruct the 3D lines. In contrast to prior solutions that primarily focused on well-behaved synthetic line drawings with severely restricting assumptions, we develop an algorithm that can work on real images. The algorithm produces line reconstruction by identifying 95% correct connectivity constraints in York Urban database, with a total computation time of 1 second per image.

Algebraic solution for the visual hull

We introduce an algebraic dual-space method for reconstructing the visual hull of a three-dimensional object from occluding contours observed in 2D images. The method exploits the differential structure of the manifold rather than parallax geometry, and therefore requires no correspondences. We begin by observing that the set of 2D contour tangents determines a surface in a dual space where each point represents a tangent plane to the original surface. The primal and dual surfaces have a symmetric algebra: A point on one is orthogonal to its dual point and tangent basis on the other. Thus the primal surface can be reconstructed if the local dual tangent basis can be estimated. Typically this is impossible because the dual surface is noisy and riddled with tangent singularities due to self-crossings. We identify a directionally-indexed local tangent basis that is well-defined and estimable everywhere on the dual surface. The estimation procedure handles singularities in the dual surface and degeneracies arising from measurement noise. The resulting method has O(N) complexity for N observed contour points and gives asymptotically exact reconstructions of surfaces that are totally observable from occluding contours.

Optimal parking in group elevator control

We consider the problem of optimally parking empty cars in an elevator group so as to anticipate and intercept the arrival of new passengers and minimize their waiting times. Two solutions are proposed, for the down-peak and up-peak traffic patterns. We demonstrate that matching the distribution of free cars to the arrival distribution of passengers is sufficient to produce savings of up to 80% in down-peak traffic. Since this approach Is not useful for the much harder case of up-peak traffic, we propose a solution based on the representation of the elevator system as a Markov decision process (MDP) model with relatively few aggregated states, and determination of the optimal parking policy by means of dynamic programming on the MDP model.

Exact calculation of expected waiting times for group elevator control

We present an efficient algorithm for exact calculation and minimization of expected waiting times of all passengers using a bank of elevators. The dynamics of the system are represented by a discrete-state Markov chain embedded in the continuous phase-space diagram of a moving elevator car. The chain is evaluated efficiently using dynamic programming to compute measures of future system performance such as expected waiting time, properly averaged over all possible future scenarios. A linear-time elevator group controller based on this method significantly outperforms benchmark algorithms and is completely within the computational capabilities of contemporary elevator bank controllers.

A conditional random field for automatic photo editing

We introduce a method for fully automatic touch-up of face images by making inferences about the structure of the scene and undesirable textures in the image. A distribution over image segmentations and labelings is computed via a conditional random field; this distribution controls the application of various local image transforms to regions in the image. Parameters governing both the labeling and transforms are jointly optimized w.r.t. a training set of before-and-after example images. One major advantage of our formulation is the ability to approximately marginalize over all possible labelings and thus exploit much or most of the information in the distribution; this yields better results than MAP inference. We demonstrate with a system that is trained to correct red-eye, reduce specularities, and remove acne and other blemishes from faces, showing results with test images scavenged from acne-themed internet message boards.

On a multiplicative update dual optimization algorithm for constrained linear MPC

We discuss a multiplicative update quadratic programming algorithm with applications to model predictive control for constrained linear systems. The algorithm, named PQP, is very simple to implement and thus verify, does not require projection, offers a linear rate of convergence, and can be completely parallelized. The PQP algorithm is equipped with conditions that guarantee the desired bound on suboptimality and with an acceleration step based on projection-free line search. We also show how PQP can take advantage of the parametric structure of the MPC problem, thus moving offline several calculations and avoiding large input/output dataflows. The algorithm is evaluated on two benchmark problems, where it is shown to compete with, and possibly outperform, other open source and commercial packages.

Routing with Probabilistic Delay Guarantees in Wireless Ad-Hoc Networks

In many wireless ad-hoc networks it is important to find a route that delivers a message to the destination within a certain deadline (delay constraint). We propose to identify such routes based on average channel state information (CSI) only, since this information can be distributed more easily over the network. Such cases allow probabilistic QoS guarantees i.e., we maximize and report the probability of on-time delivery. We develop a convolution-free lower bound on probability of on-time arrival, and a scheme to rapidly identify a path that maximizes this bound. This analysis is motivated by a class of infinite variance subexponential distributions whose properties preclude the use of deviation bounds and convolutional schemes. The bound then forms the basis of an algorithm that finds routes that give probabilistic delay guarantees. Simulations demonstrate that the algorithm performs better than shortest-path algorithm based on statistics of path loss or CSI.