Kar-Han Tan

Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging

We present a non-photorealistic rendering approach to capture and convey shape features of real-world scenes. We use a camera with multiple flashes that are strategically positioned to cast shadows along depth discontinuities in the scene. The projective-geometric relationship of the camera-flash setup is then exploited to detect depth discontinuities and distinguish them from intensity edges due to material discontinuities.We introduce depiction methods that utilize the detected edge features to generate stylized static and animated images. We can highlight the detected features, suppress unnecessary details or combine features from multiple images. The resulting images more clearly convey the 3D structure of the imaged scenes.We take a very different approach to capturing geometric features of a scene than traditional approaches that require reconstructing a 3D model. This results in a method that is both surprisingly simple and computationally efficient. The entire hardware/software setup can conceivably be packaged into a self-contained device no larger than existing digital cameras.

Specular reflection reduction with multi-flash imaging

We present a novel method to reduce the effect of specularities in digital images. Our approach relies on a simple modification of the capture setup: a multi-flash camera is used to take multiple pictures of the scene, each one with a differently positioned light source. We then formulate the problem of specular highlights reduction as solving a Poisson equation on a gradient field obtained from the input images. Experimental results are demonstrated on real and synthetic images. The entire setup can be conceivably packaged into a self-contained device, no larger than existing digital cameras.

Exploiting Depth Discontinuities for Vision-Based Fingerspelling Recognition

We present a novel method for automatic fingerspelling recognition which is able to discriminate complex hand configurations with high amounts of finger occlusions. Such a scenario, while common in most fingerspelling alphabets, presents a challenge for vision methods due to the low intensity variation along important shape edges in the hand image. Our approach is based on a simple and cheap modification of the capture setup: a multi-flash camera is used with flashes strategically positioned to cast shadows along depth discontinuities in the scene, allowing efficient and accurate hand shape extraction. We then use a shift and scale invariant shape descriptor for fingerspelling recognition, demonstrating great improvement over methods that rely on features acquired by traditional edge detection and segmentation algorithms.

Discontinuity preserving stereo with small baseline multi-flash illumination

Currently, sharp discontinuities in depth and partial occlusions in multiview imaging systems pose serious challenges for many dense correspondence algorithms. However, it is important for 3D reconstruction methods to preserve depth edges as they correspond to important shape features like silhouettes which are critical for understanding the structure of a scene. In this paper, we show how active illumination algorithms can produce a rich set of feature maps that are useful in dense 3D reconstruction. We start by showing a method to compute a qualitative depth map from a single camera, which encodes object relative distances and can be used as a prior for stereo. In a multiview setup, we show that along with depth edges, binocular half-occluded pixels can also be explicitly and reliably labeled. To demonstrate the usefulness of these feature maps, we show how they can be used in two different algorithms for dense stereo correspondence. Our experimental results show that our enhanced stereo algorithms are able to extract high quality, discontinuity preserving correspondence maps from scenes that are extremely challenging for conventional stereo methods.

Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-flash Imaging

We present a novel approach for enhancing images and video used in endoscopic surgery so that they are better able to convey shape. Our method is based on multi-flash imaging, in which multiple light sources are strategically positioned to cast shadows along depth discontinuities. We describe designs for achieving multi-flash imaging using multiple endoscopes as well as in single endoscopes. Multi-flash photography can also be used for creating medical illustrations. By highlighting the detected edges, suppressing unnecessary details, or combining features from multiple images, the resulting images convey more clearly the 3D structure of the subject. The method is easy to implement both in software and hardware, and can operate in realtime.

Multiflash Stereopsis: Depth-Edge-Preserving Stereo with Small Baseline Illumination

Traditional stereo matching algorithms are limited in their ability to produce accurate results near depth discontinuities, due to partial occlusions and violation of smoothness constraints. In this paper, we use small baseline multiflash illumination to produce a rich set of feature maps that enable the acquisition of discontinuity preserving point correspondences. First, from a single multiflash camera, we formulate a qualitative depth map using a gradient domain method that encodes object relative distances. Then, in a multiview setup, we exploit shadows created by light sources to compute an occlusion map. Finally, we demonstrate the usefulness of these feature maps by incorporating them into two different dense stereo correspondence algorithms, the first based on local search and the second based on belief propagation. Experimental results show that our enhanced stereo algorithms are able to extract high-quality discontinuity preserving correspondence maps from scenes that are extremely challenging for conventional stereo methods. We also demonstrate that small baseline illumination can be useful to handle specular reflections in stereo imagery. Different from most existing active illumination techniques, our method is simple, inexpensive, and compact and requires no calibration of light sources.

Specular highlights detection and reduction with multi-flash photography

We present a novel method to reduce the effect of specularities in digital images. Our approach relies on a simple modification of the capture setup: a multi-flash camera is used to take multiple pictures of the scene, each one with a differently positioned light source. We then formulate the problem of specular highlights reduction as solving a Poisson equation on a gradient field obtained from the input images. The obtained specular reduced image is further refined in a matting process with the maximum composite of the input images. Experimental results are demonstrated on real and synthetic images. The entire setup can be conceivably packaged into a self-contained device, no larger than existing digital cameras.

Recognition of Isolated Fingerspelling Gestures Using Depth Edges

Although steady progress has been made on developing vision-based gesture recognition systems, state-of-the-art approaches are still limited to discriminate hand configurations with high amounts of finger occlusions, a common scenario in most fingerspelling alphabets. In this article, we propose a novel method for recognition of isolated fingerspelling gestures based on depth edge features. Our approach is based on a simple and inexpensive modification of the capture setup: a multi-flash camera is used with flashes strategically positioned to cast shadows along depth discontinuities in the scene, allowing efficient and accurate extraction of depth edges. We then use a shift and scale invariant shape descriptor for fingerspelling recognition, demonstrating great improvement over methods that rely on features acquired by traditional edge detection and segmentation algorithms.

Harnessing real-world depth edges with multiflash imaging

A method for capturing geometric features of real-world scenes relies on a simple capture setup modification. The system might conceivably be packaged into a portable self-contained device. The multiflash imaging method bypasses 3D geometry acquisition and directly acquires depth edges from images. In the place of expensive, elaborate equipment for geometry acquisition, we use a camera with multiple strategically positioned flashes. Instead of having to estimate the full 3D coordinates of points in the scene (using, for example, 3D cameras) and then look for depth discontinuities, our technique reduces the general 3D problem of depth edge recovery to one of 2D intensity edge detection. Our method could, in fact, help improve current 3D cameras, which tend to produce incorrect results near depth discontinuities. Exploiting the imaging geometry for rendering provides a simple and inexpensive solution for creating stylized images from real scenes. We believe that our camera will be a useful tool for professional artists and photographers, and we expect that it will also let the average user easily create stylized imagery. This article is available with a short video documentary on CD-ROM.

The non-photorealistic camera: automatic stylization with multi-flash imaging

While photographs are the de factovisual medium for depicting reality, for some scenarios it is hard to produce pictures that convey clearly the 3D structure of a scene to the human eye. Consider imaging a white piece of paper with a white background. A traditional camera will record a mostly white image, and the shape of the paper will be lost or difficult to perceive. Our goal is to create enhanced images and video that make it easy for the viewer to understand the relative depth of the objects in the scenes depicted. This non-photorealistic camera is inspired by techniques used by skilled artists and digital illustrators to make images comprehensible: accentuating important features and reducing visual clutter. A multi-flash camera uses strategically positioned flashes to cast shadows along silhouettes in the scene, which can then be reliably detected. The detected silhouettes can then be rendered in cartoon style or as technical illustrations. This overcomes the need for perframe photo editing or 3D scanning of environments and allows automatic stylization of real world images.