Yuanzhen Li

Removing photography artifacts using gradient projection and flash-exposure sampling

Flash images are known to suffer from several problems: saturation of nearby objects, poor illumination of distant objects, reflections of objects strongly lit by the flash and strong highlights due to the reflection of flash itself by glossy surfaces. We propose to use a flash and no-flash (ambient) image pair to produce better flash images. We present a novel gradient projection scheme based on a gradient coherence model that allows removal of reflections and highlights from flash images. We also present a brightness-ratio based algorithm that allows us to compensate for the falloff in the flash image brightness due to depth. In several practical scenarios, the quality of flash/no-flash images may be limited in terms of dynamic range. In such cases, we advocate using several images taken under different flash intensities and exposures. We analyze the flash intensity-exposure space and propose a method for adaptively sampling this space so as to minimize the number of captured images for any given scene. We present several experimental results that demonstrate the ability of our algorithms to produce improved flash images.

Diffuse-Specular Separation and Depth Recovery from Image Sequences

Specular reflections present difficulties for many areas of computer vision such as stereo and segmentation. To separate specular and diffuse reflection components, previous approaches generally require accurate segmentation, regionally uniform reflectance or structured lighting. To overcome these limiting assumptions, we propose a method based on color analysis and multibaseline stereo that simultaneously estimates the separation and the true depth of specular reflections. First, pixels with a specular component are detected by a novel form of color histogram differencing that utilizes the epipolar constraint. This process uses relevant data from all the stereo images for robustness, and addresses the problem of color occlusions. Based on the Lambertian model of diffuse reflectance, stereo correspondence is then employed to compute for specular pixels their corresponding diffuse components in other views. The results of color-based detection aid the stereo correspondence, which determines both separation and true depth of specular pixels. Our approach integrates color analysis and multibaseline stereo in a synergistic manner to yield accurate separation and depth, as demonstrated by our results on synthetic and real image sequences.

Image statistics for surface reflectance perception

Human observers can distinguish the albedo of real-world surfaces even when the surfaces are viewed in isolation, contrary to the Gelb effect. We sought to measure this ability and to understand the cues that might underlie it. We took photographs of complex surfaces such as stucco and asked observers to judge their diffuse reflectance by comparing them to a physical Munsell scale. Their judgments, while imperfect, were highly correlated with the true reflectance. The judgments were also highly correlated with certain image statistics, such as moment and percentile statistics of the luminance and subband histograms. When we digitally manipulated these statistics in an image, human judgments were correspondingly altered. Moreover, linear combinations of such statistics allow a machine vision system (operating within the constrained world of single surfaces) to estimate albedo with an accuracy similar to that of human observers. Taken together, these results indicate that some simple image statistics have a strong influence on the judgment of surface reflectance.

Multiple-cue illumination estimation in textured scenes

In this paper, we present a method that integrates cues from shading, shadow and specular reflections for estimating directional illumination in a textured scene. Texture poses a problem for lighting estimation, since texture edges can be mistaken for changes in illumination condition, and unknown variations in albedo make reflectance model fitting impractical. Unlike previous works which all assume known or uniform reflectance, our method can deal with the effects of textures by capitalizing on physical consistencies that exist among the lighting cues. Since scene textures do not exhibit such coherence, we use this property to minimize the influence of texture on illumination direction estimation. For the recovered light source directions, a technique for estimating their intensities in the presence of texture is also proposed.

ScribbleBoost: adding classification to edge-aware interpolation of local image and video adjustments

One of the most common tasks in image and video editing is the local adjustment of various properties (e.g., saturation or brightness) of regions within an image or video. Edge‐aware interpolation of user‐drawn scribbles offers a less effort‐intensive approach to this problem than traditional region selection and matting. However, the technique suffers a number of limitations, such as reduced performance in the presence of texture contrast, and the inability to handle fragmented appearances. We significantly improve the performance of edge‐aware interpolation for this problem by adding a boosting‐based classification step that learns to discriminate between the appearance of scribbled pixels. We show that this novel data term in combination with an existing edge‐aware optimization technique achieves substantially better results for the local image and video adjustment problem than edge‐aware interpolation techniques without classification, or related methods such as matting techniques or graph cut segmentation.

Multibaseline stereo in the presence of specular reflections

We address the problem of accurate depth estimation using multibaseline stereo in the presence of specular reflections. Specular reflections can cause the intensity and color of corresponding points to change dramatically according to different viewpoints, thus producing severe matching errors for various stereo algorithms. In this paper we propose a new method to deal with this problem by treating specular reflections as occlusions. Our idea is to first detect specular pixels by computing the uncertainty of depth estimates. Then we combine the use of flexible windows and an adaptively selected subset of images to avoid these specular areas in all the multibaseline stereo images. Even though specularities may exist in the reference image, accurate depth is nevertheless estimated for all pixels. Experiments show that our consideration of specular reflections leads to improved stereo results.

Do colored highlights look like highlights

Case IV: When a colored specular component was combined with a white diffuse component (e.g., red on white), the surface images looked somewhat strange. They looked less glossy, and more importantly, did not appear to have a uniform reflectance. Colored highlight regions appeared to be spatially segregated from the surrounding white-body regions, as if pieces of colored foil were attached to a white matte surface.

Single-image reflectance estimation for relighting by iterative soft grouping

Reflectance values for image-based relighting are often estimated from grouped pixels with similar reflectance, but such groupings are difficult to compute with certainty for sparse image data. To address this problem, we propose an iterative method that aggregates BRDF data in a single image with known geometry and lighting by soft grouping, where pixels contribute to one anothers estimate according to their degree of reflectance similarity. Estimation of specular reflectance is further improved by albedo-independent soft grouping of pixels based on shape continuity. With recovered reflectances, we demonstrate realistic relighting for synthetic and real scenes, including surfaces with spatially-varying reflectance.

Image mapping using local and global statistics

We describe a set of techniques for mapping one image to another based on the statistics of a training set. We apply these techniques to the problems of image denoising and superresolution, but they should also be useful for many vision problems where training data are available. Given a local feature vector computed from an input image patch, we learn to estimate a subband coefficient of the output image conditioned on the patch. This entails approximating a multidimensional function, which we make tractable by nested binning and linear regression within bins. This method performs as well as nearest neighbor techniques, but is much faster. After attaining this local (patch based) estimate, we force the marginal subband histograms to match a set of target histograms, in the style of Heeger and Bergen.1 The target histograms are themselves estimated from the training data. With the combined techniques, denoising performance is similar to state of the art techniques in terms of PSNR, and is slightly superior in subjective quality. In the case of superresolution, our techniques produce higher subjective quality than the competing methods, allowing us to attain large increases in apparent resolution. Thus, for these two tasks, our method is very fast and very effective.