Andrew Lippman

Movie-maps: An application of the optical videodisc to computer graphics

An interactive, dynamic map has been built using videodisc technology to engage the user in a simulated “drive” through an unfamiliar space. The driver, or map reader, is presented with either sparsely sampled sequences of images taken by single frame cameras that replicate actual imagery from a space, or with computer synthesized replicas of those images. The reader may control the speed, route, angle of view and mode of presentation of this information and may thus tour the area. In addition, he may access spatially stored ancillary data stored in the buildings or in locales in the environment. This basic map is being enhanced to provide topographic views, and to incorporate optical and electronic image processing to provide a more responsive, visually complete representation of an environment.

Statistical models of video structure for content analysis and characterization

Content structure plays an important role in the understanding of video. In this paper, we argue that knowledge about structure can be used both as a means to improve the performance of content analysis and to extract features that convey semantic information about the content. We introduce statistical models for two important components of this structure, shot duration and activity, and demonstrate the usefulness of these models with two practical applications. First, we develop a Bayesian formulation for the shot segmentation problem that is shown to extend the standard thresholding model in an adaptive and intuitive way, leading to improved segmentation accuracy. Second, by applying the transformation into the shot duration/activity feature space to a database of movie clips, we also illustrate how the Bayesian model captures semantic properties of the content. We suggest ways in which these properties can be used as a basis for intuitive content-based access to movie libraries.

A probabilistic architecture for content-based image retrieval

The design of an effective architecture for content-based retrieval from visual libraries requires careful consideration of the interplay between feature selection, feature representation, and similarity metric. We present a solution where all the modules strive to optimize the same performance criteria: the probability of retrieval error. This solution consists of a Bayesian retrieval criteria (shown to generalize the most prevalent similarity metrics in current use) and an embedded mixture representation over a multiresolution feature space (shown to provide a good trade-off between retrieval accuracy, invariance, perceptual relevance of similarity, and complexity). The new representation extends standard models (histogram and Gaussian) by providing simultaneous support for high-dimensional features and multi-modal densities and performs well on color texture, and generic image databases.

Towards semantically meaningful feature spaces for the characterization of video content

Efficient procedures for browsing, filtering, sorting or retrieving pictorial content require accurate content characterization. Of particular interest are representations based on semantically meaningful feature spaces, capable of capturing properties such as violence, sex or profanity. In this work we report on a first step towards this goal, the design of a stochastic model for video editing which provides a transformation from the image space to a low-dimensional feature space where categorization by degree of action can be easily accomplished.

Bayesian modeling of video editing and structure: semantic features for video summarization and browsing

The ability to model content semantics is an important step towards the development of intelligent interfaces to large image and video databases. While an extremely difficult problem in the abstract, semantic characterization is possible in domains where a significant amount of structure is exhibited by the content. Whenever this is the case, given their ability to integrate prior knowledge about this structure in the inferences to be made, Bayesian methods are a natural solution to the problem. We present a Bayesian architecture for content characterization and analyze its potential as a tool for accessing and browsing through video databases on a semantic basis.

Energy-efficient cooperative routing in multi-hop wireless ad hoc networks

We study the routing problem for multi-hop wireless ad hoc networks based on cooperative transmission. We prove that the minimum energy cooperative path (MECP) routing problem, i.e., using cooperative radio transmission to find the best route with the minimum energy cost from a source node to a destination node, is NP-complete. We thus propose a cooperative shortest path (CSP) algorithm that uses the Dijkstras algorithm as the basic building block and reflects the cooperative transmission properties in the relaxation procedure. Simulation results show that with more nodes added in the network, our approach achieves more energy saving compared to traditional non-cooperative shortest path algorithms. Another interesting observation is that the proposed algorithm achieves better fairness among different nodes with denser networks. Implementation issues are also discussed

Models for automatic classification of video sequences

In this paper, we explore a technique for automatic classification of video sequences, (such as a TV broadcast, movies). This technique analyzes the incoming video sequences and classifies them into categories. It can be viewed as an on-line parser for video signals. We present two techniques for automatic classification. In the first technique, the incoming video sequence is analyzed to extract the motion information. This information is optimally projected onto a single dimension. This projection information is then used to train Hidden Markov Models (HMMs) that efficiently and accurately classify the incoming video sequence. Preliminary results with 50 different test sequences (25 Sports and 25 News sequences) indicae a classification accuracy of 90% by the HMM models. In the second technique, 24 full-length motion picture trailers are classified using HMMs. This classification is compared with the internet movie database and we find that they correlate well. Only two out of 24 trailers were classified incorrectly.

Spatio-temporal segmentation based on motion and static segmentation

The problem of segmenting an image sequence in terms of regions characterized by a coherent motion is among the most challenging in image sequence analysis. This paper proposes a new technique which sequentially refines the segmentation and the motion estimation by combining static segmentation and motion information. The motion is robustly computed by a global estimation which remove the camera motion, followed by a local estimation using a matching technique and a robust estimator. Simulation results show the efficiency of the proposed technique.

Cooperative diversity with opportunistic relaying

In this paper, we present single-selection-opportunistic-relaying with decode-and-forward (DaF) and amplify-and-forward (AaF) protocols under an aggregate power constraint. We show that opportunistic DaF relaying is equivalent to the outage bound of the optimal DaF strategy using all potential relays. We further show that opportunistic AaF relaying is outage-optimal with single-relay selection and significantly outperforms an AaF strategy with multiple-relay (MR) transmissions, in the presence of limited channel knowledge. These findings reveal that cooperative diversity benefits (under an aggregate power constraint) are useful even when cooperative relays choose not to transmit but rather choose to cooperatively listen; they act as passive relays and give priority to the transmission of a single opportunistic relay

A spatiotemporal motion model for video summarization

The compact description of a video sequence through a single image map and a dominant motion has applications in several domains, including video browsing and retrieval, compression, mosaicing, and visual summarization. Building such a representation requires the capability to register all the frames with respect to the dominant object in the scene, a task which has been, in the past, addressed through temporally localized motion estimates. In this paper, we show how the lack of temporal consistency associated with such estimates can undermine the validity of the dominant motion assumption, leading to oscillation between different scene interpretations and poor registration. To avoid this oscillation, we augment the motion model with a generic temporal constraint which increases the robustness against competing interpretations, leading to more meaningful content summarization.