Seung-Ryong Han

Time-Varying Mesh Compression Using an Extended Block Matching Algorithm

Time-varying mesh, which is attracting a lot of attention as a new multimedia representation method, is a sequence of 3-D models that are composed of vertices, edges, and some attribute components such as color. Among these components, vertices require large storage space. In conventional 2-D video compression algorithms, motion compensation (MC) using a block matching algorithm is frequently employed to reduce temporal redundancy between consecutive frames. However, there has been no such technology for 3-D time-varying mesh so far. Therefore, in this paper, we have developed an extended block matching algorithm (EBMA) to reduce the temporal redundancy of the geometry information in the time-varying mesh by extending the idea of the 2-D block matching algorithm to 3-D space. In our EBMA, a cubic block is used as a matching unit. MC in the 3-D space is achieved efficiently by matching the mean normal vectors calculated from partial surfaces in cubic blocks, which our experiments showed to be a suboptimal matching criterion. After MC, residuals are transformed by the discrete cosine transform, uniformly quantized, and then encoded. The extracted motion vectors are also entropy coded after differential pulse code modulation. As a result of our experiments, 10%-18% compression has been achieved.

A High Quality Depth Map Upsampling Method Robust to Misalignment of Depth and Color Boundaries

In recent years, fusion camera systems that consist of color cameras and Time-of-Flight (TOF) depth sensors have been popularly used due to its depth sensing capability at real-time frame rates. However, captured depth maps are limited in low resolution compared to the corresponding color images due to physical limitation of the TOF depth sensor. Most approaches to enhancing the resolution of captured depth maps depend on the implicit assumption that when neighboring pixels in the color image have similar values, they are also similar in depth. Although many algorithms have been proposed, they still yield erroneous results, especially when region boundaries in the depth map and the color image are not aligned. We therefore propose a novel kernel regression framework to generate the high quality depth map. Our proposed filter is based on the vector pointing similar pixels that represents the unit vector toward similar neighbors in the local region. The vectors are used to detect misaligned regions between color edges and depth edges. Unlike conventional kernel regression methods, our method properly handles misaligned regions by introducing the numerical analysis of the local structure into the kernel regression framework. Experimental comparisons with other data fusion techniques prove the superiority of the proposed algorithm.

Geometry compression for time-varying meshes using coarse and fine levels of quantization and run-length encoding

Time-varying meshes (TVM) is a new 3-D scene representation which are generated from multiple cameras. It captures highly detailed shape and texture as well as movement of real-world moving objects. Compression is a key technology for supporting TVM applications such as education, interactive broadcasting, and intangible heritage archiving. Previous works focused on compression of 3-D animation that has the same topology throughout the entire sequences. Unfortunately, the topology of TVMs change with time which makes it difficult to compress TVMs. In this paper, we propose a geometry encoder for TVMs. The encoder finds spatial and temporal redundancy by coarse and fine level quantization. Thereafter, vertex information is converted into binary sequences. And then, the binary sequences are encoded using run-length encoding (RLE). Experimental results show that vertices of TVMs which require 96 bits per vertex (bpv) are compressed to 1.9-15.4 bpv while maintaining a small geometric distortion ranging from 0.7 times 10-4 to 1.3 times 10-3 % of the maximum error.

3D Video Compression Based on Extended Block Matching Algorithm

Three dimensional (3D) video is attracting a lot of attention as a new multimedia representation method. 3D video is a sequence of 3D models (frames) that consist of varying vertices and connectivity. In conventional 2D video compression algorithms, motion compensation (MC) using block matching algorithm is frequently employed to reduce redundancy between consecutive frames. However, there is no such technology for 3D video so far. Therefore, in this paper, we have developed an extended block matching algorithm (EMBA) to reduce temporal redundancy of geometry information of 3D video by extending the idea of 2D block matching to 3D space. In our EBMA, a cubic block is used as a matching unit and, MC is achieved efficiently by matching the mean normal vectors of the sub-blocks, which turned out to be sub-optimal by our experiments. The residual information is further transformed by discrete cosine transform (DCT) and then encoded. The extracted motion vectors are also entropy encoded. As a result of our experiments, compression ratio ranging from 10% to 18% of the original 3D video data has been achieved.

Automatic preview video generation for mesh sequences

We present a novel method that automatically generates a preview video of a mesh sequence. To make the preview appealing to users, the important features of the mesh model should be captured in the preview video, while preserving the constraint that the transitions of the camera are as smooth as possible. Our approach models the important features by defining a surface saliency and by measuring the appearance of the mesh sequence. The task of generating the preview video is then formulated as a shortest-path problem and we find an optimal camera path by using Dijkstras algorithm.

Trilateral filter construction for depth map upsampling

In recent years, fusion camera systems that consist of color cameras and Time-of-Flight (TOF) depth sensors have been popularly used due to its depth sensing capability at real-time frame rates. However, captured depth maps are limited in low resolution compared to the corresponding color images due to physical limitation of the TOF depth sensor. Although many algorithms have been proposed, they still yield erroneous results, especially when boundaries of the depth map and the color image are not aligned. We therefore propose a novel kernel regression framework to generate the high quality depth map. Our proposed filter is based on the vector pointing homogeneous pixels that represents the unit vector toward similar neighbors in the local region. The vectors are used to detect misaligned regions between color edges and depth edges. Experimental comparisons with other data fusion techniques prove the superiority of the proposed algorithm.

Photometric and geometric rectification for stereoscopic images

Stereoscopic images are captured by two cameras at different positions. In general, the two images often have geometric distortions including vertical misalignment, rotation and keystone as well as photometric distortions such as luminance or color differences. Even in case of a carefully designed parallel stereo camera configuration, the captured images pairs may have the distortions that cause uncomfortable 3D experiences to users. In this paper, we develop an algorithm to correct the captured image pairs to give a comfortable stereoscopic experience to users. The algorithm provides a practical method for compensating the photometric and geometrical distortions.

Approaches to 3D video compression

Three-dimensional (3-D) video provides an immersing experience for users. In recent years, many attempts have been made to capture the complex surface shape and highly detailed texture of real-world moving objects, which results in a huge amount of data. In this paper, we discuss compression issues for 3-D video. We introduce 3-D video, which is classified into two categories. Then we survey compression methods that have been investigated for each category. We present our compression methods for temporally varying mesh sequences. In addition, we show comparison results for our algorithm with respect to previous work.

Mixed spatial and SNR scalability for TVM geometry coding

Time Varying Meshes (TVMs) are sequences of three-dimensional models created from multi-camera images. Like other kinds of mesh sequences, it has a great need for compression, in order to become largely usable. Alongside with other types of signals, mesh coding has received considerable attention towards scalable coding, which is a very important feature in many kinds of coded signals. Scalable mesh coding can be divided basically into three types of scalability: temporal, spatial and SNR (signal-to-noise ratio). In this work we focus on the use of block-based and run-length encoding aiming at achieving a combination of spatial and SNR scalability for the geometrical information of TVM. Our results have shown that it is possible to achieve scalability with only a minor increase in rate over a previous coding method.