Ketan Mayer-Patel

Coordinated multi-streaming for 3D tele-immersion

This paper looks at the problem of multi-streaming in 3D tele-immersion and describes how a protocol architecture called CP (for Coordination Protocol) can be used to coordinate video frame transport between application clusters. CP provides application endpoints with information about current network conditions, and an open architecture for implementing application-specific coordination schemes. The scheme described in this paper apportions available bandwidth among flows such that frame transport synchrony, important for 3D reconstruction performance, is significantly enhanced. Results demonstrating the effectiveness of CP in increasing multi-stream coordination, while at the same time maintaining aggregate congestion responsiveness, are obtained from a FreeBSD/Linux implementation and a live experimental network. Results underscore the importance of consistency in network information across flows for realizing dramatic improvements in frame arrival synchrony.

An MPEG performance model and its application to adaptive forward error correction

We present a general analytical model for predicting the reconstructed frame rate of an MPEG stream. Our model captures the temporal relationships between I-, P, and B-frames but is independent of the channel and media characteristics. We derive an adaptive FEC scheme from the general model and verify it by comparing it to the results of a simulation. The prediction error of the model compared to the simulation for a wide array of parameter values is less than 5%. We then use the derived adaptive FEC scheme to study the optimal rate allocation (i.e., between generating a higher frame rate or increasing the protection for a lower frame rate) when equation-based TCP rate control is used to couple packet rates to channel characteristics such as round trip time and packet loss probabilities. Surprisingly, we find that optimal protection levels for I- and P-frames are relatively static as loss rates increase from 1% to 4% while changes in the frame type pattern are used to ameliorate the effects of the increased loss. The study demonstrates how our model can be used to reveal joint source/channel coding tradeoffs and how they relate to encoding and transmission parameters.

Real-time compression for dynamic 3D environments

The goal of tele-immersion has long been to enable people at remote locations to share a sense of presence. A tele-immersion system acquires the 3D representation of a collaborators environment remotely and sends it over the network where it is rendered in the users environment. Acquisition, reconstruction, transmission, and rendering all have to be done in real-time to create a sense of presence. With added commodity hardware resources, parallelism can increase the acquisition volume and reconstruction data quality while maintaining real-time performance. However this is not as easy for rendering since all of the data need to be combined into a single display.In this paper we present an algorithm to compress data from such 3D environments in real-time to solve this imbalance. We expect the compression algorithm to scale comparably to the acquisition and reconstruction, reduce network transmission bandwidth, and reduce the rendering requirement for real-time performance. We have tested the algorithm using a synthetic office data set and have achieved a 5 to 1 compression for 22 depth streams.

Synchronized continuous media playback through the World Wide Web

We have developed and demonstrated a remote continuous media playback application called cmplayer, using the Berkeley Continuous Media Toolkit (CMT). Cmp layer can be used in conjunction with aWorld W]de Web browserto access remotely stored continuous media data (i.e., video and audio) and provide synchronized playback of multiple media streams. The cmplayer communicates witha server process also written with CMT to stream media data across the network using adaptive control to compensate forlimitednetwork bandwidth, server utilization, and client CPU resources. Asimple script format isusedto indicate thelocation of each piece of media data (i.e., hostname, port number and the filename) and specify source synchronization between media streams. Multiple video and audio streams can be specified. In addition, scrolling text streams and streams of Tcl commands can be integrated as stream types. A stream can be stored in multiple media data files distributed on different servers. Cmplayer script files are ASCII text and are typically less than 10 lines long. Cmplayer provides full VCR-like controls and random access into any point of the video.

Real-time multidepth stream compression

The goal of tele-immersion has long been to enable people at remote locations to share a sense of presence. A tele-immersion system acquires the 3D representation of a collaborators environment remotely and sends it over the network where it is rendered in the users environment. Acquisition, reconstruction, transmission, and rendering all have to be done in real-time to create a sense of presence. With added commodity hardware resources, parallelism can increase the acquisition volume and reconstruction data quality while maintaining real-time performance. However, this is not as easy for rendering since all of the data need to be combined into a single display.In this article, we present an algorithm to compress data from such 3D environments in real-time to solve this imbalance. We present a compression algorithm which scales comparably to the acquisition and reconstruction, reduces network transmission bandwidth, and reduces the rendering requirement for real-time performance. This is achieved by exploiting the coherence in the 3D environment data and removing them in real-time. We have tested the algorithm using a static office data set as well as a dynamic scene, the results of which are presented in the article.

A general framework for multidimensional adaptation

Data adaptation is an essential system component in a wide variety of application areas. To date, most applications use ad hoc methods to manage data in response to limited resources and changing system conditions. We present a generic adaptation framework that distills the common elements essential to a broad class of adaptive applications. Our framework provides an abstract data representation and defines a generic set of adaptation operations that directly support multidimensional and multimedia adaptive behavior. We present several application case studies and demonstrate the performance of our framework on an experimental prototype.

dc : A live webcast control system

Live Internet streaming media programs, called webcasts, can adopt techniques developed by television to obtain higher quality. We have developed a general webcast production model composed of three stages (i.e., sources, broadcast, and transmission) and a tool, called the Directors Console (dc), to control live webcasts. The tool is one component of a distributed service architecture, which adapts to varying physical infrastructure and broadcast configurations.

Design and performance of the Berkeley Continuous Media Toolkit

The design and performance of the Berkeley Continuous Media Toolkit (CMT) is described. CMT provides a programming environment for rapid development of continuous media applications. CMT overhead is measured in the context of a simple video playback application and is found to be only a few milliseconds per frame played. As a demonstration of CMT as a research infrastructure, an experiment comparing adaptive frame rate control policies is described.

Spatially-encoded far-field representations for interactive walkthroughs

We introduce the notion of spatially encoded video and use it for efficiently representing image-based impostors for interactive walkthroughs. As part of a pre-process, we automatically decompose the model and compute the far-fields. The resulting texture images are organized along multiple dimensions and can be accessed in a user-steered order at interactive rates. Our encoding algorithm can compress the impostors size by two orders of magnitude. Furthermore, the storage cost for additional impostors or samples grows sub-linearly. The resulting system has been applied to a complex CAD environment composed of 13 million triangles. We are able to render it at interactive rates on a PC with little loss in image quality.

Exploiting temporal parallelism for software-only video effects processing

hternet video is emerging as an important mtitimedia ap pHcation area. Video effects are important dernents in conventional video, but they are tiequently used in hternet ~ideo. Current hard~~based video effects proc~sing SG lutions ~e not w~ matched for the ~+abti@ of kternet video (ie., bit rates, packet loss, jitter, etc.). A SO*+ ody solution, in contrast, provides the fl~ubtity to match this &abti@. The key to a sohe solution is to ~\Tloit pardeh~m. This paper presents the design of a pardel soti+ordy video Meets processing system and pretinary e~~erirnentd r~ts that mTlore the use of temporal pardl&~m.