King To Ng

The compression of simplified dynamic light fields

This paper studies the compression of a dynamic image-based rendering (IBR) representation called simplified dynamic light fields (SDLF). It is obtained by constraining the viewpoints in a dynamic environment along a line instead of a 2D plane. The SDLFs have a dimensionality of four, which considerably simplifies their capture and data compression. A new coding algorithm for SDLFs using a modified MPEG-2 algorithm is proposed. It employs both temporal and spatial predictions from the reference video streams to explore better the redundancy among the light field images. Experimental results, using a synthetic SDLF, show that the proposed compression scheme offers a 2 dB improvement in PSNR over a similar coding scheme using only temporal prediction.

The plenoptic video

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic video. It is a simplified light field for dynamic environments, where user viewpoints are constrained to the camera plane of a linear array of video cameras. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. The system consists of a camera array of eight Sony CCX-Z11 CCD cameras and eight Pentium 4 1.8-GHz computers connected together through a 100 Base-T local area network. It is possible to perform software-assisted real-time MPEG-2 compression at a resolution of (720/spl times/480). Using selective transmission, we are able to stream continuously plenoptic video with (256/spl times/256) resolution at a rate of 15 f/s over the network. For rendering from raw data on the hard disk, real-time rendering can be achieved with a resolution of (720/spl times/480) and a rate of 15 f/s. A new compression algorithm using both temporal and spatial predictions is also proposed for the efficient compression of the plenoptic videos. Experimental results demonstrate the usefulness of the proposed parallel processing based system in capturing and rendering high-quality dynamic image-based representations using off-the-shelf equipment, and its potential applications in visualization and immersive television systems.

An Object-Based Approach to Image/Video-Based Synthesis and Processing for 3-D and Multiview Televisions

This paper proposes an object-based approach to a class of dynamic image-based representations called ldquoplenoptic videos,rdquo where the plenoptic video sequences are segmented into image-based rendering (IBR) objects each with its image sequence, depth map, and other relevant information such as shape and alpha information. This allows desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects to be supported. Moreover, the rendering quality in scenes with large depth variations can also be improved considerably. A portable capturing system consisting of two linear camera arrays was developed to verify the proposed approach. An important step in the object-based approach is to segment the objects in video streams into layers or IBR objects. To reduce the time for segmenting plenoptic videos under the semiautomatic technique, a new object tracking method based on the level-set method is proposed. Due to possible segmentation errors around object boundaries, natural matting with Bayesian approach is also incorporated into our system. Furthermore, extensions of conventional image processing algorithms to these IBR objects are studied and illustrated with examples. Experimental results are given to illustrate the efficiency of the tracking, matting, rendering, and processing algorithms under the proposed object-based framework.

The plenoptic videos: capturing, rendering and compression

This paper presents a system for capturing and rendering a dynamic image-based representation called the plenoptic videos. It is a simplified version of light fields for dynamic environment, where user viewpoints are constrained along the camera plane of a linear array of video cameras. The system consists of a camera array of 8 Sony CCX-Z11 CCD cameras and eight Pentium 41.8 GHz computers connected together through a 100 baseT LAN. Important issues such as multiple camera calibration, real-time compression, decompression and rendering are addressed. Experimental results demonstrated the usefulness of the proposed parallel processing based system in capturing and rendering high quality dynamic image-based representation using off-the-shelf equipment, and its potential applications in visualization and immersive television systems.

Virtual reality using the concentric mosaic: construction, rendering and data compression

This paper proposes a new image based rendering technique called concentric mosaic for virtual reality applications. It is constructed by capturing vertical slit images when a camera is moving around a set of concentric circles. Concentric mosaic allows the user to move freely in a circular region and observe significant parallax and lighting changes without recovering the geometric and photometric scene model. The rendering of concentric mosaic is very efficient, which amounts to reordering and interpolating of previously captured slit images in the concentric mosaic. Concentric mosaic typically consists of hundreds of high-resolution images, which consumes significant amount of storage and bandwidth for transmission. An MPEG-like compression algorithm is therefore proposed taking advantages of the access patterns and redundancies of the mosaic images. Experimental results show that real-time reconstruction of novel views with good image quality can be achieved in a Pentium II 300 MHz PC.

Bi-level video: video communication at very low bit rates

The rapid development of wired and wireless networks tremendouslyfacilitates communications between people. However, most of thecurrent wireless networks still work in low bandwidths, and mobiledevices still suffer from weak computational power, short batterylifetime and limited display capability. We developed a very lowbit-rate bi-level video coding technique, which can be used invideo communications almost anywhere, anytime on any device. Thespirit of this method is that rather than giving highest priorityto the basic colors of an image as in conventional DCT-basedcompression methods, we give preference to the outline features ofscenes when we have limited bandwidths. These features can berepresented by bi-level image sequences that are converted fromgray-scale image sequences. By analyzing the temporal correlationbetween successive frames and flexibilities in the scenepresentation using bi-level images, we achieve very high ratioswith our bi-level video compression scheme. Experiments show thatin low bandwidths, our method provides clearer shape, smoothermotion, shorter initial latency and much cheaper computational costthan do DCT-based methods. Our method is especially suitable forsmall mobile devices such as handheld PCs, palm-size PCs and mobilephones that possess small display screens and light computationalpower, and work in low bandwidth wireless networks. We have builtPC and Pocket PC versions of bi-level video phone systems, whichtypically provide QCIF-size video with a frame rate of 5-15 fps fora 9.6 Kbps bandwidth.

Am object-based approach to plenoptic videos

This paper proposes an object-based approach to plenoptic videos, where the plenoptic video sequences are segmented into image-based rendering (IBR) objects each with its image sequence, depth map and other relevant information such as shape information. This allows desirable functionalities such as scalability of contents, error resilience, and interactivity with individual IBR objects to be supported. A portable capturing system consisting of two linear camera arrays, each hosting 6 JVC video cameras, was developed to verify the proposed approach. Rendering and compression results of real-world scenes demonstrate the usefulness and good quality of the proposed approach.

Buffer control algorithm for low bit-rate video compression

In this paper, a new buffer control algorithm for motion-compensated hybrid DPCM/DCT coding (like H.261 and MPEG-1 I pictures) is presented. The algorithm uses the bit allocation algorithm to determine the quantization scale factor of each macroblock under a given target bit rate. An important advantage of the algorithm is that it has precise control of the buffer and avoids buffer overflow events which is a severe problem in low bit rate video coders. Furthermore, the coder is able to allocate bits to the picture as a whole, resulting in better rate-distortion trade-off. Simulation results show that the H.261 coder, using the proposed algorithm, can achieve a higher PSNR and better visual quality than a codec using a conventional buffer control algorithm.

Object-Based Coding for Plenoptic Videos

A new object-based coding system for a class of dynamic image-based representations called plenoptic videos (PVs) is proposed. PVs are simplified dynamic light fields, where the videos are taken at regularly spaced locations along line segments instead of a 2-D plane. In the proposed object-based approach, objects at different depth values are segmented to improve the rendering quality. By encoding PVs at the object level, desirable functionalities such as scalability of contents, error resilience, and interactivity with an individual image-based rendering (IBR) object can be achieved. Besides supporting the coding of texture and binary shape maps for IBR objects with arbitrary shapes, the proposed system also supports the coding of grayscale alpha maps as well as depth maps (geometry information) to respectively facilitate the matting and rendering of the IBR objects. Both temporal and spatial redundancies among the streams in the PV are exploited to improve the coding performance, while avoiding excessive complexity in selective decoding of PVs to support fast rendering speed. Advanced spatial/temporal prediction methods such as global disparity-compensated prediction, as well as direct prediction and its extensions are developed. The bit allocation and rate control scheme employing a new convex optimization-based approach are also introduced. Experimental results show that considerable improvements in coding performance are obtained for both synthetic and real scenes, while supporting the stated object-based functionalities.

A Multi-Camera Approach to Image-Based Rendering and 3-D/Multiview Display of Ancient Chinese Artifacts

This paper proposes an image-based approach for the capturing, rendering and display of ancient Chinese artifacts for cultural heritage preservation. A multiple-camera circular array is proposed to record images of the artifacts, which forms a simplified circular light field (SCLF). A systematic image-based approach and associate algorithms such as segmentation, depth estimation and shape morphing are developed for rendering new views of the Chinese artifacts. An object-based compression scheme is also proposed to reduce the data size for storage and transmission of the texture, depth maps and alpha maps associated with the object-based circular light field. Spatial redundancies among the various images are exploited to improve the coding performance, while avoiding excessive complexity in selective decoding of the light field to support fast rendering speed. To allow the Chinese artifacts to be viewed over the internet, scalable prioritized transmission and rendering schemes of the SCLF with low latency were also developed. The multiple views so synthesized enable the ancient artifacts to be displayed in 3-D/multi-view displays. Several collections from the University Museum and Art Gallery at The University of Hong Kong were captured and excellent rendering results are obtained.