Yuanyuan Ding

Importance filtering for image retargeting

Content-aware image retargeting has attracted a lot of interests recently. The key and most challenging issue for this task is how to balance the tradeoff between preserving the important contents and minimizing the visual distortions on the consistency of the image structure. In this paper we present a novel filtering-based technique to tackle this issue, called ”importance filtering”. Specifically, we first filter the image saliency guided by the image itself to achieve a structure-consistent importance map. We then use the pixel importance as the key constraint to compute the gradient map of pixel shifts from the original resolution to the target. Finally, we integrate the shift gradient across the image using a weighted filter to construct a smooth shift map and render the target image. The weight is again controlled by the pixel importance. The two filtering processes enforce to maintain the structural consistency and yet preserve the important contents in the target image. Furthermore, the simple nature of filter operations allows highly efficient implementation for real-time applications and easy extension to video retargeting, as the structural constraints from the original image naturally convey the temporal coherence between frames. The effectiveness and efficiency of our importance filtering algorithm are confirmed in extensive experiments.

Dynamic fluid surface acquisition using a camera array

Acquiring dynamic 3D fluid surfaces is a challenging problem in computer vision. Single or stereo camera based solutions are sensitive to refraction distortions, fast fluid motions, and calibration errors. In this paper, we present a multi-view based solution for robustly capturing fast evolving fluid wavefronts. We first construct a portable, 3×3 camera array system as the main acquisition device. We elaborately design the system to allow high-resolution and high-speed capture. To recover fluid surfaces, we place a known pattern beneath the surface and position the camera array on top to observe the pattern. By tracking the distorted feature points over time and across cameras, we obtain spatial-temporal correspondence maps and we use them for specular carving to reconstruct the time-varying surface. In case one of the cameras loses track due to distortions or blurs, we use the rest of the cameras to construct the surface and then apply multi-perspective warping to locate the lost-track feature points so that we can continue using the camera in later frames. Our experiments on synthetic and real data demonstrate that our multi-view framework is robust and reliable.

Design and Estimation of Coded Exposure Point Spread Functions

We address the problem of motion deblurring using coded exposure. This approach allows for accurate estimation of a sharp latent image via well-posed deconvolution and avoids lost image content that cannot be recovered from images acquired with a traditional shutter. Previous work in this area has used either manual user input or alpha matting approaches to estimate the coded exposure Point Spread Function (PSF) from the captured image. In order to automate deblurring and to avoid the limitations of matting approaches, we propose a Fourier-domain statistical approach to coded exposure PSF estimation that allows us to estimate the latent image in cases of constant velocity, constant acceleration, and harmonic motion. We further demonstrate that previously used criteria to choose a coded exposure PSF do not produce one with optimal reconstruction error, and that an additional 30 percent reduction in Root Mean Squared Error (RMSE) of the latent image estimate can be achieved by incorporating natural image statistics.

Analysis of motion blur with a flutter shutter camera for non-linear motion

Motion blurs confound many computer vision problems. The fluttered shutter (FS) camera [1] tackles the motion deblurring problem by emulating invertible broadband blur kernels. However, existing FS methods assume known constant velocity motions, e.g., via user specifications. In this paper, we extend the FS technique to general 1D motions and develop an automatic motion-from-blur framework by analyzing the image statistics under the FS. We first introduce a fluttered-shutter point-spread-function (FS-PSF) to uniformly model the blur kernel under general motions. We show that many commonly used motions have closed-form FS-PSFs. To recover the FS-PSF from the blurred image, we present a new method by analyzing image power spectrum statistics. We show that the Modulation Transfer Function of the 1D FS-PSF is statistically correlated to the blurred image power spectrum along the motion direction. We then recover the FS-PSF by finding the motion parameters that maximize the correlation. We demonstrate our techniques on a variety of motions including constant velocity, constant acceleration, and harmonic rotation. Experimental results show that our method can automatically and accurately recover the motion from the blurs captured under the fluttered shutter.

Modeling Complex Unfoliaged Trees from a Sparse Set of Images

We present a novel image‐based technique for modeling complex unfoliaged trees. Existing tree modeling tools either require capturing a large number of views for dense 3D reconstruction or rely on user inputs and botanic rules to synthesize natural‐looking tree geometry. In this paper, we focus on faithfully recovering real instead of realistically‐looking tree geometry from a sparse set of images. Our solution directly integrates 2D/3D tree topology as shape priors into the modeling process. For each input view, we first estimate a 2D skeleton graph from its matte image and then find a 2D skeleton tree from the graph by imposing tree topology. We develop a simple but effective technique for computing the optimal 3D skeleton tree most consistent with the 2D skeletons. For each edge in the 3D skeleton tree, we further apply volumetric reconstruction to recover its corresponding curved branch. Finally, we use piecewise cylinders to approximate each branch from the volumetric results. We demonstrate our framework on a variety of trees to illustrate the robustness and usefulness of our technique.

Recovering specular surfaces using curved line images

We present a new shape-from-distortion framework for recovering specular (reflective/refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection/refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces.

Recovering shape characteristics on near-flat specular surfaces

We consider the problem of capturing shape characteristics on specular (refractive and reflective) surfaces that are nearly flat. These surfaces are difficult to model using traditional methods based on reconstructing the surface positions and normals. These lower-order shape attributes provide little information to identify important surface characteristics related to distortions. In this paper, we present a framework for recovering the higher-order geometry attributes of specular surfaces. Our method models local reflections and refractions in terms of a special class of multiperspective cameras called the general linear cameras (GLCs). We then develop a new theory that correlates the higher-order differential geometry attributes with the local GLCs. Specifically, we show that Gaussian and mean curvature can be directly derived from the camera intrinsics of the local GLCs. We validate this theory on both synthetic and real-world specular surfaces. Our method places a known pattern in front of a reflective surface or beneath a refractive surface and captures a distorted image on the surface. We then compute the optimal GLC using a sparse set of correspondences and recover the curvatures from the GLC. Experiments demonstrate that our methods are robust and highly accurate.

Multiperspective stereo matching and volumetric reconstruction

Stereo matching and volumetric reconstruction are the most explored 3D scene recovery techniques in computer vision. Many existing approaches assume perspective input images and use the epipolar constraint to reduce the search space and improve the accuracy. In this paper we present a novel framework that uses multi-perspective cameras for stereo matching and volumetric reconstruction. Our approach first decomposes a multi-perspective camera into piecewise primitive General Linear Cameras or GLCs [32]. A pair of GLCs in general do not satisfy the epipolar constraint. However, they still form a nearly stereo pair. We develop a new Graph-Cut-based algorithm to account for the slight vertical parallax using the GLC ray geometry. We show that the recovered pseudo disparity map conveys important depth cues analogous to perspective stereo matching. To more accurately reconstruct a 3D scene, we develop a new multi-perspective volumetric reconstruction method. We discretize the scene into voxels and apply the GLC back-projections to map the voxel onto each input multi-perspective camera. Finally, we apply the graph-cut algorithm to optimize the 3D embedded voxel graph. We demonstrate our algorithms on both synthetic and real multi-perspective cameras. Experimental results show that our methods are robust and reliable.

A theory of multi-perspective defocusing

We present a novel theory for characterizing defocus blurs in multi-perspective cameras such as catadioptric mirrors. Our approach studies how multi-perspective ray geometry transforms under the thin lens. We first use the General Linear Cameras (GLCs) [21] to approximate the incident multi-perspective rays to the lens and then apply a Thin Lens Operator (TLO) to map an incident GLC to the exit GLC. To study defocus blurs caused by the GLC rays, we further introduce a new Ray Spread Function (RSF) model analogous the Point Spread Function (PSF). While PSF models defocus blurs caused by a 3D scene point, RSF models blurs spread by rays. We derive closed form RSFs for incident GLC rays, and we show that for catadioptric cameras with a circular aperture, the RSF can be effectively approximated as a single or mixtures of elliptic-shaped kernels. We apply our method for predicting defocus blurs on commonly used catadioptric cameras and for reducing de-focus blurs in catadioptric projections. Experiments on synthetic and real data demonstrate the accuracy and general applicability of our approach.

Epsilon Stereo Pairs.

Human stereo vision works by fusing a pair of perspective images with a purely horizontal parallax. Recent developments suggest that very few varieties of multiperspective stereo pairs exist. In this paper, we introduce a new stereo model, which we call epsilon stereo pairs, for fusing a broader class of multiperspective images. An epsilon stereo pair consists of two images with a slight vertical parallax. We show many multiperspective camera pairs that do not satisfy the stereo constraint can still form epsilon stereo pairs. We then introduce a new ray-space warping algorithm to minimize stereo inconsistencies in an epsilon pair using multiperspective collineations. This makes epsilon stereo model a promising tool for synthesizing close-to-stereo fusions from many non-stereo pairs.