James D. K. Kim

Interpolator data compression for MPEG-4 animation

Interpolator representation in key-frame animation is now the most popular method for computer animation. The interpolator data consist of key and key value pairs, where a key is a time stamp and a key value is the corresponding value to the key. In this paper, we propose a set of new technologies to compress the interpolator data. The performance of the proposed technique is compared with the existing MPEG-4 generic compression tool. Throughout the core experiments in MPEG-4, the proposed technique showed its superiority over the existing tool, becoming a part of MPEG-4 standard within the Animation Framework eXtension framework.

A CMOS Image Sensor Based on Unified Pixel Architecture With Time-Division Multiplexing Scheme for Color and Depth Image Acquisition

We propose a CMOS image sensor with time-division multiplexing pixel architecture using standard pinned-photodiode for capturing 2-D color image as well as extracting 3-D depth information of a target object. The proposed pixel can alternately provide both color and depth images in each frame. Two split photodiode and four transfer gates in each pixel improve the transfer speed of generated electrons to be capable of demodulating a high-frequency time-of-flight signal. In addition, four-shared pixel architecture acquires a color image with high spatial resolution and generates a reliable depth map by inherent binning operation in charge domain. A 712 496 pixel array has been fabricated using a 0.11μm standard CMOS imaging process and fully characterized. A 6m pixel with 34.5% aperture ratio can be operated at 10-MHz modulation frequency with 70% demodulation contrast. We have successfully captured both images of exactly same scene from the fabricated test chip. It shows a depth uncertainty of less than 60 mm and a linearity error of about 2% between 1 and 3 m distance with 50-ms integration time. Moreover, high-gain readout operation enables to improve the performance, achieving about 43-mm depth uncertainty at 3 m.

A Three-Dimensional Time-of-Flight CMOS Image Sensor With Pinned-Photodiode Pixel Structure

A pixel architecture for providing not only normal 2-D images but also depth information by using a conventional pinned photodiode is presented. This pixel architecture allows the sensor to generate a real-time 3-D image of an arbitrary object. The operation of the pixel is based on the time-of-flight principle detecting the time delay between the emitted and reflected infrared light pulses in a depth image mode. The pixel contains five transistors. Compared to the conventional 4-T CMOS image sensor, the new pixel includes an extra optimized transfer gate for high-speed charge transfer. A fill factor of more than 60% is achieved with a 12 × 12 μm2 size for increasing the sensitivity. A fabricated prototype sensor successfully captures 64 × 16 depth images between 1 and 4 m at a 5-MHz modulation frequency. The depth inaccuracy is measured under 2% at 1 m and 4% at 4 m and is verified by noise analysis.

Real-time facial animation from live video tracking

This paper describes a complete pipe-line of a practical system for producing real-time facial expressions of a 3D virtual avatar controlled by an actors live performances. The system handles practical challenges arising from markerless expression captures from a single conventional video camera. For robust tracking, a localized algorithm constrained by belief propagation is applied to the upper face, and an appearance matching technique using a parameterized generic face model is exploited for lower face and head pose tracking. The captured expression features then transferred to high dimensional 3D animation controls using our facial expression space which is a structure-preserving map between two algebraic structures. The transferred animation controls drive facial animation of a 3D avatar while optimizing the smoothness of the face mesh. An example-based face deformation technique produces non-linear local detail deformations on the avatar that are not captured in the movement of the animation controls.

A novel 2.5D pattern for extrinsic calibration of tof and camera fusion system

Recently, many researchers have made efforts for accurate calibration of a Time-of-Flight camera to fully utilize its provided depth values. Yet most previous works focus mainly on intrinsic calibration by modeling its systematic errors and noises while extrinsic calibration is also an important factor when constructing sensor fusion system. In this paper, we present a calibration process that can correctly transfer the depth measurements onto the color image. We use 2.5D pattern so that sufficient reprojection error can be considered for both color and ToF cameras. The issues on obtaining the correct correspondences for this pattern are discussed. In the optimization stage, the depth constraint is also employed to ensure the depth measurements to lie on the pattern plane. The strengths of the proposed method over previous approaches are evaluated in several robotic applications which require precise ToF and camera calibration.

Animation data compression in MPEG-4: interpolators

In this paper, we propose new technologies to compress the interpolator nodes in VRML/MPEG-4 BIFS (binary format for scene). Interpolators are used to animate the 3D objects in a VRML/MPEG-4 BIFS scene. For interactive applications with quite a few animated 3D synthetic objects, two components are dominant in the amount of data to represent the animation: 3D object and its animation. In the current MPEG-4, 3D model coding (3DMC) yields a high compression ratio with reasonable quality for the 3D object, while predictive MFField coding (PMFC) yields a moderate compression ratio with reasonable quality for the animation data. The proposed techniques are to provide a high compression on animation data and have been adopted to MPEG-4 system amendment 4 (AFX/MUW).

Time-of-flight sensor and color camera calibration for multi-view acquisition

This paper presents a multi-view acquisition system using multi-modal sensors, composed of time-of-flight (ToF) range sensors and color cameras. Our system captures the multiple pairs of color images and depth maps at multiple viewing directions. In order to ensure the acceptable accuracy of measurements, we compensate errors in sensor measurement and calibrate multi-modal devices. Upon manifold experiments and extensive analysis, we identify the major sources of systematic error in sensor measurement and construct an error model for compensation. As a result, we provide a practical solution for the real-time error compensation of depth measurement. Moreover, we implement the calibration scheme for multi-modal devices, unifying the spatial coordinate for multi-modal sensors.The main contribution of this work is to present the thorough analysis of systematic error in sensor measurement and therefore provide a reliable methodology for robust error compensation. The proposed system offers a real-time multi-modal sensor calibration method and thereby is applicable for the 3D reconstruction of dynamic scenes.

Range unfolding for Time-of-Flight depth cameras

Time-of-Flight depth cameras provide a direct way to acquire range images, using the phase delay of the incoming reflected signal with respect to the emitted signal. These cameras, however, have a challenging problem called range folding, which occurs due to the modular error in phase delay—ranges are modulo the maximum range. To our best knowledge, we exploit the first approach to estimate the number of mods at each pixel from only a single range image. The estimation is recasted into an optimization problem in the Markov random field framework, where the number of mods is considered as a label. The actual range is then recovered using the optimal number of mods at each pixel, so-named range unfolding. As demonstrated in the experiments with various range images of real scenes, the proposed method accurately determines the number of mods. In result, the maximum range is practically extended at least twice of that specified by the modulation frequency.

Parametric model-based noise reduction for ToF depth sensors

This paper presents a novel Time-of-Flight (ToF) depth denoising algorithm based on parametric noise modeling. ToF depth image includes space varying noise which is related to IR intensity value at each pixel. By assuming ToF depth noise as additive white Gaussian noise, ToF depth noise can be modeled by using a power function of IR intensity. Meanwhile, nonlocal means filter is popularly used as an edge-preserving denoising method for removing additive Gaussian noise. To remove space varying depth noise, we propose an adaptive nonlocal means filtering. According to the estimated noise, the search window and weighting coefficient are adaptively determined at each pixel so that pixels with large noise variance are strongly filtered and pixels with small noise variance are weakly filtered. Experimental results demonstrate that the proposed algorithm provides good denoising performance while preserving details or edges compared to the typical nonlocal means filtering.

Progressive compression of 3D triangular meshes using topology-based Karhunen-Loève transform

In this work, we propose a progressive compression algorithm using topology-based Karhunen-Loeve transform(KLT). First, we simplify an input mesh to represents an original mesh in several level of details. Then, coordinates of decimated vertices at each level are predicted from the coarser level mesh, and the prediction residuals are transmitted to a decoder. To provide high coding efficiency, we apply the topology-based KLT, which compacts the energy into a few coefficients, to the prediction residuals. Moreover, we develop a bit plane coder, which uses a context-adaptive arithmetic coder, for the entropy coding. Experiments on various 3D meshes show that the proposed algorithm provides enhanced compression performance.