Shuaicheng Liu

Seamless Video Stitching from Hand-held Camera Inputs

Images/videos captured by portable devices (e.g., cellphones, DV cameras) often have limited fields of view. Image stitching, also referred to as mosaics or panorama, can produce a wide angle image by compositing several photographs together. Although various methods have been developed for image stitching in recent years, few works address the video stitching problem. In this paper, we present the first system to stitch videos captured by hand‐held cameras. We first recover the 3D camera paths and a sparse set of 3D scene points using CoSLAM system, and densely reconstruct the 3D scene in the overlapping regions. Then, we generate a smooth virtual camera path, which stays in the middle of the original paths. Finally, the stitched video is synthesized along the virtual path as if it was taken from this new trajectory. The warping required for the stitching is obtained by optimizing over both temporal stability and alignment quality, while leveraging on 3D information at our disposal. The experiments show that our method can produce high quality stitching results for various challenging scenarios.

Joint Video Stitching and Stabilization From Moving Cameras

In this paper, we extend image stitching to video stitching for videos that are captured for the same scene simultaneously by multiple moving cameras. In practice, videos captured under this circumstance often appear shaky. Directly applying image stitching methods for shaking videos often suffers from strong spatial and temporal artifacts. To solve this problem, we propose a unified framework in which video stitching and stabilization are performed jointly. Specifically, our system takes several overlapping videos as inputs. We estimate both inter motions (between different videos) and intra motions (between neighboring frames within a video). Then, we solve an optimal virtual 2D camera path from all original paths. An enlarged field of view along the virtual path is finally obtained by a space-temporal optimization that takes both inter and intra motions into consideration. Two important components of this optimization are that: 1) a grid-based tracking method is designed for an improved robustness, which produces features that are distributed evenly within and across multiple views and 2) a mesh-based motion model is adopted for the handling of the scene parallax. Some experimental results are provided to demonstrate the effectiveness of our approach on various consumer-level videos and a Plugin, named “Video Stitcher” is developed at Adobe After Effects CC2015 to show the processed videos.

A Hierarchical Approach for Rain or Snow Removing in a Single Color Image

In this paper, we propose an efficient algorithm to remove rain or snow from a single color image. Our algorithm takes advantage of two popular techniques employed in image processing, namely, image decomposition and dictionary learning. At first, a combination of rain/snow detection and a guided filter is used to decompose the input image into a complementary pair: 1) the low-frequency part that is free of rain or snow almost completely and 2) the high-frequency part that contains not only the rain/snow component but also some or even many details of the image. Then, we focus on the extraction of image’s details from the high-frequency part. To this end, we design a 3-layer hierarchical scheme. In the first layer, an overcomplete dictionary is trained and three classifications are carried out to classify the high-frequency part into rain/snow and non-rain/snow components in which some common characteristics of rain/snow have been utilized. In the second layer, another combination of rain/snow detection and guided filtering is performed on the rain/snow component obtained in the first layer. In the third layer, the sensitivity of variance across color channels is computed to enhance the visual quality of rain/snow-removed image. The effectiveness of our algorithm is verified through both subjective (the visual quality) and objective (through rendering rain/snow on some ground-truth images) approaches, which shows a superiority over several state-of-the-art works.In this paper, we propose an efficient algorithm to remove rain or snow from a single color image. Our algorithm takes advantage of two popular techniques employed in image processing, namely, image decomposition and dictionary learning. At first, a combination of rain/snow detection and a guided filter is used to decompose the input image into a complementary pair: 1) the low-frequency part that is free of rain or snow almost completely and 2) the high-frequency part that contains not only the rain/snow component but also some or even many details of the image. Then, we focus on the extraction of images details from the high-frequency part. To this end, we design a 3-layer hierarchical scheme. In the first layer, an overcomplete dictionary is trained and three classifications are carried out to classify the high-frequency part into rain/snow and non-rain/snow components in which some common characteristics of rain/snow have been utilized. In the second layer, another combination of rain/snow detection and guided filtering is performed on the rain/snow component obtained in the first layer. In the third layer, the sensitivity of variance across color channels is computed to enhance the visual quality of rain/snow-removed image. The effectiveness of our algorithm is verified through both subjective (the visual quality) and objective (through rendering rain/snow on some ground-truth images) approaches, which shows a superiority over several state-of-the-art works.

A Hybrid Approach for Near-Range Video Stabilization

Near-range videos contain objects that are close to the camera. These videos often contain discontinuous depth variation (DDV), which is the main challenge to the existing video stabilization methods. Traditionally, 2D methods are robust to various camera motions (e.g., quick rotation and zooming) under scenes with continuous depth variation (CDV). However, in the presence of DDV, they often generate wobbled results due to the limited ability of their 2D motion models. Alternatively, 3D methods are more robust in handling near-range videos. We show that, by compensating rotational motions and ignoring translational motions, near-range videos can be successfully stabilized by 3D methods without sacrificing the stability too much. However, it is time-consuming to reconstruct the 3D structures for the entire video and sometimes even impossible due to rapid camera motions. In this paper, we combine the advantages of 2D and 3D methods, yielding a hybrid approach that is robust to various camera motions and can handle the near-range scenarios well. To this end, we automatically partition the input video into CDV and DDV segments. Then, the 2D and 3D approaches are adopted for CDV and DDV clips, respectively. Finally, these segments are stitched seamlessly via a constrained optimization. We validate our method on a large variety of consumer videos.

Kernel-free video deblurring via synthesis

Document is unavailable: This DOI was registered to an article that was not presented by the author(s) at this conference. As per section 8.2.1.B.13 of IEEEs Publication Services and Products Board Operations Manual, IEEE has chosen to exclude this article from distribution. We regret any inconvenience.

CodingFlow: Enable Video Coding for Video Stabilization

Video coding focuses on reducing the data size of videos. Video stabilization targets at removing shaky camera motions. In this paper, we enable video coding for video stabilization by constructing the camera motions based on the motion vectors employed in the video coding. The existing stabilization methods rely heavily on image features for the recovery of camera motions. However, feature tracking is time-consuming and prone to errors. On the other hand, nearly all captured videos have been compressed before any further processing and such a compression has produced a rich set of block-based motion vectors that can be utilized for estimating the camera motion. More specifically, video stabilization requires camera motions between two adjacent frames. However, motion vectors extracted from video coding may refer to non-adjacent frames. We first show that these non-adjacent motions can be transformed into adjacent motions such that each coding block within a frame contains a motion vector referring to its adjacent previous frame. Then, we regularize these motion vectors to yield a spatially-smoothed motion field at each frame, named as CodingFlow, which is optimized for a spatially-variant motion compensation. Based on CodingFlow, we finally design a grid-based 2D method to accomplish the video stabilization. Our method is evaluated in terms of efficiency and stabilization quality, both quantitatively and qualitatively, which shows that our method can achieve high-quality results compared with the state-of-the-art methods (feature-based).

Joint bundled camera paths for stereoscopic video stabilization

This paper presents a method to stabilize shaky stereoscopic videos captured by hand-held devices. Directly applying traditional monocular video stabilization techniques to two views independently is problematic as it often brings undesirable vertical disparities and produces inaccurate horizontal disparities, which violate original stereoscopic disparity constraints, leading to erroneous depth perception. In this paper, we show that monocular video stabilization methods, such as the bundled camera paths stabilization, can be extended for stereoscopic videos by taking additional disparity constraints during the stabilization. In particular, we first estimate disparities between two views. Then, we compute camera motions as meshes of bundled paths for each view. Next, we smooth paths of two views separately and iteratively. During each iteration, we adjust the meshes of one view by our proposed `Joint Disparity and Stability mesh Warp (JDSW). The final result is generated after several iterations of paths smoothing and meshes adjusting, in which temporal stability and correct depth perception are achieved simultaneously. We evaluate our method by various challenging stereoscopic videos with different camera motions and scene types. The experiments demonstrate the effectiveness of our method.

Cross-Space Distortion Directed Color Image Compression

Traditional color image compression is usually conducted in the YCbCr space but many color displayers only accept RGB signals as inputs. Due to the use of a non-unitary matrix in the YCbCr-RGB conversion, low distortion achieved in the YCbCr space cannot guarantee low distortion for the RGB signals. To solve this problem, we propose a novel compression scheme for color images through defining a cross-space distortion so as to reduce as much as possible the distortion in the RGB space. To this end, we first derive the relationship between the distortions in the YCbCr space and RGB space. Then, we develop two solutions to implement color image compression for the most popular 4:2:0 chroma format. The first solution focuses on the design of a new spatial downsampling method to generate the 4:2:0 YCbCr image for a high-efficiency compression. The second one provides a novel way to reduce the distortion of the compressed color image by controlling the quantization error of the 4:2:0 YCbCr image, especially the one generated by using the traditional spatial downsampling. Experimental results show that both proposed solutions offer a remarkable quality gain over some state-of-the-art approaches when tested on various textured color images.

Long-Distance/Environment Face Image Enhancement Method for Recognition

With the increase of distance and the influence of environmental factors, such as illumination and haze, the face recognition accuracy is significantly lower than that of indoor close-up images. In order to solve this problem, an effective face image enhancement method is proposed in this paper. This algorithm is a nonlinear transformation which combines gamma and logarithm transformation. Therefore, it is called: G-log. The G-Log algorithm can perform the following functions: (1) eliminate the influence of illumination; (2) increase image contrast and equalize histogram; (3) restore the high-frequency components and detailed information; (4) improve visual effect; (5) enhance recognition accuracy. Given a probe image, the procedure of face alignment, enhancement and matching is executed against all gallery images. For comparing the effects of different enhancement algorithms, all probe images are processed by different enhancement methods and identical face alignment, recognition modules. Experiment results show that G-Log method achieves the best effect both in matching accuracy and visual effect. Long-distance uncontrolled environment face recognition accuracy has been greatly improved, up to 98%, 98%, 95% for 60-, 100-, 150-m images after processed by G-Log from original 95%, 89%, 70%.

Uncertain Region Identification for Stereoscopic Foreground Cutout

This paper presents a method that automatically segments the foreground objects for stereoscopic images. Given a stereo pair, a disparity map can be estimated, which encodes the depth information. Objects that stay close to the camera are considered as foreground while regions with larger depths are deemed as background. Although the raw disparity map is usually noisy, incomplete, and inaccurate, it facilitates an automatic generation of trimaps for both views, where the images are partitioned into three regions: definite foreground, definite background, and uncertain region. Our job is now reduced to labelling of pixels in the uncertain region, in which the number of unknown pixels has been decreased largely. We propose to use an MRF based energy minimization for labelling the unknown pixels, which involves both local and global color probabilities within and across views. Results are evaluated by objective metrics on a ground truth stereo segmentation dataset, which validates the effectiveness of our proposed method.