Fu Li

Structure guided fusion for depth map inpainting

Depth acquisition becomes inexpensive after the revolutionary invention of Kinect. For computer vision applications, depth maps captured by Kinect require additional processing to fill up missing parts. However, conventional inpainting methods for color images cannot be applied directly to depth maps as there are not enough cues to make accurate inference about scene structures. In this paper, we propose a novel fusion based inpainting method to improve depth maps. The proposed fusion strategy integrates conventional inpainting with the recently developed non-local filtering scheme. The good balance between depth and color information guarantees an accurate inpainting result. Experimental results show the mean absolute error of the proposed method is about 20mm, which is comparable to the precision of the Kinect sensor.

An efficient VLSI architecture for 4×4 intra prediction in the High Efficiency Video Coding (HEVC) standard

Intra prediction with fine directions is a critical feature in the new High Efficiency Video Coding (HEVC) standard because it provides significant performance gain. Different from the intra prediction in the H.264/AVC, this approach is more complicated in terms of computation and memory access, which makes the VLSI design very difficult. In this paper, we propose an efficient uniform architecture for all of the 4×4 intra directional modes. The architecture is implemented by a register array and a flexible reference sample selection technique. This novel architecture does not need to project the samples from the side reference to the main reference. Thus, it reduces the processing latency and the number of registers considerably. The proposed architecture has been implemented with TSMC 0.13μm CMOS technology. Simulation results show that the proposed architecture only needs 9020 logic gates for 17 directional modes and can run at 150 MHz operation frequency.

Pattern Masking Estimation in Image With Structural Uncertainty

A model of visual masking, which reveals the visibility of stimuli in the human visual system (HVS), is useful in perceptual based image/video processing. The existing visual masking function mainly considers luminance contrast, which always overestimates the visibility threshold of the edge region and underestimates that of the texture region. Recent research on visual perception indicates that the HVS is sensitive to orderly regions that possess regular structures and insensitive to disorderly regions that possess uncertain structures. Therefore, structural uncertainty is another determining factor on visual masking. In this paper, we introduce a novel pattern masking function based on both luminance contrast and structural uncertainty. Through mimicking the internal generative mechanism of the HVS, a prediction model is firstly employed to separate out the unpredictable uncertainty from an input image. In addition, an improved local binary pattern is introduced to compute the structural uncertainty. Finally, combining luminance contrast with structural uncertainty, the pattern masking function is deduced. Experimental result demonstrates that the proposed pattern masking function outperforms the existing visual masking function. Furthermore, we extend the pattern masking function to just noticeable difference (JND) estimation and introduce a novel pixel domain JND model. Subjective viewing test confirms that the proposed JND model is more consistent with the HVS than the existing JND models.

Square wave encoded fringe patterns for high accuracy depth sensing

In this paper, we propose a square wave encoded in three sinusoidal fringe patterns for depth sensing. The periods of the square wave and the sinusoidal wave are coprime. Because of the specific pattern design strategy, a Gabor filter is used to obtain the wrapped phase, and the coprime theorem is utilized to reliably determine the absolute phase of the encoded sinusoidal wave. Quantitative analyses and practical experiments have been presented to verify the performance of the proposed method. The precision of our method is close to that of the classic three-step phase-shifting method. Besides, depth of discontinuous surfaces can be correctly measured. By adopting a high-resolution projector and camera, our proposed method can acquire a denser and more precise depth map than an original Kinect and ToF camera.

A low complexity multiplierless transform coding for HEVC

Transform coding with multiple blocks results in high complexity of HEVC. Butterfly combined with multipliers method provides an efficient implementation. However, the matrix multiplication is inevitable based on this method. In this paper, matrices and element decomposition are proposed to decompose the multiplication matrices into orthogonal matrices and general matrices with smaller elements. The number of different elements in these matrices can be reduced after decomposing, so that the multiplications can be reduced. Further, the decomposed elements become smaller and much closer to power of 2, which means we can use shifting and adding to achieve the multiplications conveniently. The proposed method is suitable for the hardware implementation.

One-shot depth acquisition with a random binary pattern

In this paper, we propose a new spatial encoding method that integrates the random binary pattern and the improved phase-difference-matching method to acquire a dense and precise depth map. The adopted binary pattern can simplify pattern projecting devices compared with the patterns that use color. The density of speckles in the pattern is periodic and the positions of them are random. Based on these two properties, we propose an improved phase-difference corresponding method, which is divided into two steps: the coarse matching step to estimate the approximate coordinates of pixels in the pattern via analyzing the phase values of the image, and the fine matching step to compensate errors of the coarse matching results and to achieve subpixel accuracy. This matching method does not require an extra optimization method with high computational complexity. In the experiment, we show the effectiveness of the proposed method. We also evaluate this method in actual experiments. The results show that this method has advantages over the time-of-flight camera and Kinect, particularly in terms of precision.

Single Shot Dual-Frequency Structured Light Based Depth Sensing

Structured light techniques are widely used for depth sensing. In this paper, we propose a single shot dual-frequency structured light based method to achieve dense depth in dynamic scenes. The projected pattern is a mixture of two different periodical waves whose phases are related to the change of color and intensity respectively, which can avoid the requirement of Fourier spectra separation in other multi-frequency patterns. Gabor filter is adopted to interpolate the phases. The number theory is used to deal with the phase ambiguity in phase based method conveniently and speedily. A dense depth can be achieved because of the phase-based encoding mode. The proposed method is suitable for dense depth acquisition of the moving object. Experimental results show higher accuracy of the proposed method in depth acquisition compared with the Kinect and larger resolution compared with the ToF (Time of Flight) depth camera. Meanwhile, the proposed method can also acquire the depth of the color scenes and is robust to the surface texture of objects.

Maximum a posteriori-based depth sensing with a single-shot maze pattern

This study addressed the general problem of correspondence retrieval for single-shot depth sensing where the coded features cannot be detected perfectly. The traditional correspondence retrieval technique can be regarded as maximum likelihood estimation with a uniform distribution prior assumption, which may lead to mismatches for two types of insignificant features: 1) incomplete features that cannot be detected completely because of edges, tiny objects, and many depth variations, etc.; and 2) distorted features disturbed by environmental noise. To overcome the drawback of the uniform distribution assumption, we propose a maximum a posteriori estimation-based correspondence retrieval method that uses the significant features as priors to estimate the weak or missing features. We also propose a novel monochromatic maze-like pattern, which is more robust to ambient illumination and the colors in scenes than the traditional patterns. Our experimental results demonstrate that the proposed system performs better than the popular RGB-D cameras and traditional single-shot techniques in terms of accuracy and robustness, especially with challenging scenes.

Single-shot dense depth sensing with frequency-division multiplexing fringe projection

Abstract In structured light illumination (SLI) systems, multi-shot fringe patterns can reach higher precision than a single-shot fringe pattern. However, multi-shot methods are not suitable for dynamic scenes while single-shot ones are limited in the measurement accuracy. In this paper, a novel single-shot depth sensing method with frequency-division multiplexing (FDM) framework is proposed. To achieve a simultaneous casting, two fringe patterns with coprime periods are modulated into a single pattern. The method of fringe pattern extraction is similar to the demodulation in communication systems. The Gabor filter is adopted to get the phase information in the pattern, and the coprime theorem is used to solve the phase ambiguity. Quantitative and qualitative evaluations have proved that our method achieves higher accuracy in depth sensing compared with the Kinect v1 and ToF camera. In addition, benefiting from the single-shot pattern, our method is suitable for dynamic scenes.

High quality impulse noise removal via non-uniform sampling and autoregressive modelling based super-resolution.

The challenge of image impulse noise removal is to restore spatial details from damaged pixels using remaining ones in random locations. Most existing methods use all uncontaminated pixels within a local window to estimate the centred noisy one via a statistic way. These kinds of methods have two defects. First, all noisy pixels are treated as independent individuals and estimated by their neighbours one by one, with the correlation between their true values ignored. Second, the image structure as a natural feature is usually ignored. This study proposes a new denoising framework, in which all noisy pixels are jointly restored via non-uniform sampling and supervised piecewise autoregressive modelling based super-resolution. In this method, the noisy pixels are jointly estimated in groups through solving a well-designed optimisation problem, in which image structure feature is considered as an important constraint. Another contribution is that piecewise autoregressive model is not simply adopted but carefully designed so that all noise-free pixels can be used to supervise the model training and optimisation problem solving for higher accuracy. The experimental results demonstrate that the proposed method exhibits good denoising performance in a large noise density range (10–90%).