Seung-Won Jung

Directional Joint Bilateral Filter for Depth Images

Depth maps taken by the low cost Kinect sensor are often noisy and incomplete. Thus, post-processing for obtaining reliable depth maps is necessary for advanced image and video applications such as object recognition and multi-view rendering. In this paper, we propose adaptive directional filters that fill the holes and suppress the noise in depth maps. Specifically, novel filters whose window shapes are adaptively adjusted based on the edge direction of the color image are presented. Experimental results show that our method yields higher quality filtered depth maps than other existing methods, especially at the edge boundaries.

A consensus-driven approach for structure and texture aware depth map upsampling.

This paper presents a method for increasing spatial resolution of a depth map using its corresponding high-resolution (HR) color image as a guide. Most of the previous methods rely on the assumption that depth discontinuities are highly correlated with color boundaries, leading to artifacts in the regions where the assumption is broken. To prevent scene texture from being erroneously transferred to reconstructed scene surfaces, we propose a framework for dividing the color image into different regions and applying different methods tailored to each region type. For the region classification, we first segment the low-resolution (LR) depth map into regions of smooth surfaces, and then use them to guide the segmentation of the color image. Using the consensus of multiple image segmentations obtained by different super-pixel generation methods, the color image is divided into continuous and discontinuous regions: in the continuous regions, their HR depth values are interpolated from LR depth samples without exploiting the color information. In the discontinuous regions, their HR depth values are estimated by sequentially applying more complicated depth-histogram-based methods. Through experiments, we show that each step of our method improves depth map upsampling both quantitatively and qualitatively. We also show that our method can be extended to handle real data with occluded regions caused by the displacement between color and depth sensors.

Image Contrast Enhancement Using Color and Depth Histograms

In this letter, we propose a new global contrast enhancement algorithm using the histograms of color and depth images. On the basis of the histogram-modification framework, the color and depth image histograms are first partitioned into sub-intervals using the Gaussian mixture model. The positions partitioning the color histogram are then adjusted such that spatially neighboring pixels with the similar intensity and depth values can be grouped into the same sub-interval. By estimating the mapping curve of the contrast enhancement for each sub-interval, the global image contrast can be improved without over-enhancing the local image contrast. Experimental results demonstrate the effectiveness of the proposed algorithm.

Adaptive post-filtering of JPEG compressed images considering compressed domain lossless data hiding

In this paper, we propose a new postprocessing algorithm for JPEG compressed images considering a compressed domain lossless data hiding scheme. The conventional compressed domain lossless data hiding scheme can embed information bits into the bitstream, while maintaining the file size and JPEG compatibility. As a new application of the conventional scheme, we design an algorithm of embedding filter coefficients in the bitstream, such that the embedded filter can be used to enhance the quality of the decoded image. To this end, a Wiener filter is estimated between the original and decoded images, and the coefficients are embedded into the JPEG bitstream. Moreover, the size, symmetry, and precision of the filter and the number of filters are determined under the constraint of embeddable bits. The proposed JPEG decoder extracts the filter coefficients from the bitstream and enhances the quality of the decoded image by applying the Wiener filter. The proposed algorithm can also work with conventional JPEG postprocessing algorithms, so that the performance of such conventional algorithms can be further improved.

Color and Depth Image Correspondence for Kinect v2

Kinect v2, a new version of Kinect sensor, provides RGB, IR (Infrared) and depth images like its predecessor Kinect v1. However, the depth measurement mechanism and the image resolutions of the Kinect v2 are different from those of Kinect v1, which requires a new transformation matrix for the camera calibration of Kinect v2. In this paper, we correct the radial distortion of the RGB camera and find the transformation matrix for the correspondence between the RGB and depth image of the Kinect v2. Experimental results show that our method yields accurate correspondence between the RGB and depth images.

Rotated top-bottom dual-kinect for improved field of view

Existing commodity depth sensors have limited the field of view (FOV) of depth scanning. Our solution for extending the FOV is to use multiple depth sensors and stitch the captured depth images to a depth panorama. In our case study, we use two Kinects to address the following two questions: what is the best layout of the two Kinects to maximize the FOV and, second, how to combine the depth images together to form the depth panorama. We answer these questions by proposing a rotated top-bottom (RTB) arrangement of the two Kinects to maximize the FOV. Since the two Kinects capture the depth images from their own views, the depth values are not necessarily identical for the same object. To solve this problem, the depth adjustments are made for a frontal reference coordinate. Moreover, the perspective distortions of the two Kinects with respect to the frontal reference coordinate are corrected by perspective transformations. Experimental results show that our RTB sensor can generate panorama depth images with an almost doubled FOV.

Order-Preserving Condensation of Moving Objects in Surveillance Videos

Vision-based detection of illegal or accidental activities in urban traffic has attracted great interest. Since state-of-the-art online automated detection algorithms are far from perfect, much research effort on offline video surveillance has been made to prevent police or security staff from observing all recorded video frames unnecessarily. To solve the problem, this study focuses on video condensation, which provides fast monitoring of moving objects in a long duration of surveillance videos. Considering the computational complexity and the condensation ratio as the two main criteria for efficient video condensation, we propose a video condensation algorithm, which consists of the following: 1) initial condensation by discarding frames of nonmoving objects; 2) intra-GoFM (group of frames with moving objects) condensation; and 3) inter-GoFM condensation. In the intra-GoFM and inter-GoFM condensation, spatiotemporal static pixels within each GoFM and temporal static pixels between two consecutive GoFMs are dropped to shorten the temporal distances between consecutive moving objects. Experimental results show that our video condensation saves a significant amount of computational loads compared with the previous methods without sacrificing the condensation ratio and visual quality.

Depth completion for kinect v2 sensor

Kinect v2 adopts a time-of-flight (ToF) depth sensing mechanism, which causes different type of depth artifacts comparing to the original Kinect v1. The goal of this paper is to propose a depth completion method, which is designed especially for the Kinect v2 depth artifacts. Observing the specific types of depth errors in the Kinect v2 such as thin hole-lines along the object boundaries and the new type of holes in the image corners, in this paper, we exploit the position information of the color edges extracted from the Kinect v2 sensor to guide the accurate hole-filling around the object boundaries. Since our approach requires a precise registration between color and depth images, we also introduce the transformation matrix which yields point-to-point correspondence with a pixel-accuracy. Experimental results demonstrate the effectiveness of the proposed depth image completion algorithm for the Kinect v2 in terms of completion accuracy and execution time.

Lossless embedding of depth hints in JPEG compressed color images

The conventional JPEG compressed domain lossless data hiding scheme has three attractive properties: file size is maintained, decoded image is unchanged, and embedded bitstream is compatible with the JPEG standard. This study introduces a new application of the compressed domain lossless data hiding scheme. Specifically, we present an algorithm that embeds depth hints to a compressed bitstream of a color image, which enables the end-user to extract depth hints and reconstruct the depth image. On the limited watermark capacity, depth hints are obtained using super-pixel color image segmentation and depth value clustering. The experimental results demonstrate the effectiveness of the proposed algorithm. HighlightsA compressed-domain lossless data hiding scheme is adopted to embed depth hints.Joint color and depth image processing is used to extract depth hints.The best possible depth hints are found under the watermark capacity constraint.The proposed method enables the end-user to reconstruct depth images.A promising image refocusing application is shown.

Exact Histogram Specification Considering the Just Noticeable Difference

Exact histogram specification (EHS) transforms the histogram of an input image into the specified histogram. In the conventional EHS techniques, the pixels are first sorted according to their graylevels, and the pixels that have the same graylevel are further differentiated according to the local average of the pixel values and the edge strength. The strictly ordered pixels are then mapped to the desired histogram. However, since the conventional sorting method is inherently dependent on the initial graylevel-based sorting, the contrast enhancement capability of the conventional EHS algorithms is restricted. We propose a modified EHS algorithm considering the just noticeable difference. In the proposed algorithm, the edge pixels are pre-processed such that the output edge pixels obtained by the modified EHS can result in the local contrast enhancement. Moreover, we introduce a new sorting method for the pixels that have the same graylevel. Experimental results show that the proposed algorithm provides better image enhancement performance compared to the conventional EHS algorithms.