Seon Joo Kim

Detailed Real-Time Urban 3D Reconstruction from Video

Abstract The paper presents a system for automatic, geo-registered, real-time 3D reconstruction from video of urban scenes. The system collects video streams, as well as GPS and inertia measurements in order to place the reconstructed models in geo-registered coordinates. It is designed using current state of the art real-time modules for all processing steps. It employs commodity graphics hardware and standard CPU’s to achieve real-time performance. We present the main considerations in designing the system and the steps of the processing pipeline. Our system extends existing algorithms to meet the robustness and variability necessary to operate out of the lab. To account for the large dynamic range of outdoor videos the processing pipeline estimates global camera gain changes in the feature tracking stage and efficiently compensates for these in stereo estimation without impacting the real-time performance. The required accuracy for many applications is achieved with a two-step stereo reconstruction process exploiting the redundancy across frames. We show results on real video sequences comprising hundreds of thousands of frames.

Robust Radiometric Calibration and Vignetting Correction

In many computer vision systems, it is assumed that the image brightness of a point directly reflects the scene radiance of the point. However, the assumption does not hold in most cases due to nonlinear camera response function, exposure changes, and vignetting. The effects of these factors are most visible in image mosaics and textures of 3D models where colors look inconsistent and notable boundaries exist. In this paper, we propose a full radiometric calibration algorithm that includes robust estimation of the radiometric response function, exposures, and vignetting. By decoupling the effect of vignetting from the response function estimation, we approach each process in a manner that is robust to noise and outliers. We verify our algorithm with both synthetic and real data, which shows significant improvement compared to existing methods. We apply our estimation results to radiometrically align images for seamless mosaics and 3D model textures. We also use our method to create high dynamic range (HDR) mosaics that are more representative of the scene than normal mosaics.

A New In-Camera Imaging Model for Color Computer Vision and Its Application

We present a study of in-camera image processing through an extensive analysis of more than 10,000 images from over 30 cameras. The goal of this work is to investigate if image values can be transformed to physically meaningful values, and if so, when and how this can be done. From our analysis, we found a major limitation of the imaging model employed in conventional radiometric calibration methods and propose a new in-camera imaging model that fits well with todays cameras. With the new model, we present associated calibration procedures that allow us to convert sRGB images back to their original CCD RAW responses in a manner that is significantly more accurate than any existing methods. Additionally, we show how this new imaging model can be used to build an image correction application that converts an sRGB input image captured with the wrong camera settings to an sRGB output image that would have been recorded under the correct settings of a specific camera.

Constructing image panoramas using dual-homography warping

This paper describes a method to construct seamless image mosaics of a panoramic scene containing two predominate planes: a distant back plane and a ground plane that sweeps out from the cameras location. While this type of panorama can be stitched when the camera is carefully rotated about its optical center, such ideal scene capture is hard to perform correctly. Existing techniques use a single homography per image to perform alignment followed by seam cutting or image blending to hide inevitable alignments artifacts. In this paper, we demonstrate how to use two homographies per image to produce a more seamless image. Specifically, our approach blends the homographies in the alignment procedure to perform a nonlinear warping. Once the images are geometrically stitched, they are further processed to blend seams and reduce curvilinear visual artifacts due to the nonlinear warping. As demonstrated in our paper, our procedure is able to produce results for this type of scene where current state-of-the-art techniques fail.

Visual enhancement of old documents with hyperspectral imaging

Hyperspectral imaging (HSI) of historical documents is becoming more common at national libraries and archives. HSI is useful for many tasks related to document conservation and management as it provides detailed quantitative measurements of the spectral reflectance of the document that is not limited to the visible spectrum. In this paper, we focus on how to use the invisible spectra, most notably near-infrared (NIR) bands, to assist in visually enhancing old documents. Specifically, we demonstrate how to use the invisible bands to improve the visual quality of text-based documents corrupted with undesired artifacts such as ink-bleed, ink-corrosion, and foxing. For documents of line drawings that suffer from low contrast, we use details found in the invisible bands to enhance legibility. The key components of our framework involve detecting regions in the document that can be enhanced by the NIR spectra, compositing the enhanced gradient map using the NIR bands, and reconstructing the final image from the composited gradients. This work is part of a collaborative effort with the Nationaal Archief of the Netherlands (NAN) and Art Innovation, a manufacturer of hyperspectral imaging hardware designed specially for historical documents. Our approach is evaluated on historical documents from NAN that exhibit degradations common to documents found in most archives and libraries.

Color Transfer Using Probabilistic Moving Least Squares

This paper introduces a new color transfer method which is a process of transferring color of an image to match the color of another image of the same scene. The color of a scene may vary from image to image because the photographs are taken at different times, with different cameras, and under different camera settings. To solve for a full nonlinear and nonparametric color mapping in the 3D RGB color space, we propose a scattered point interpolation scheme using moving least squares and strengthen it with a probabilistic modeling of the color transfer in the 3D color space to deal with mis-alignments and noise. Experiments show the effectiveness of our method over previous color transfer methods both quantitatively and qualitatively. In addition, our framework can be applied for various instances of color transfer such as transferring color between different camera models, camera settings, and illumination conditions, as well as for video color transfers.

Nonlinear Camera Response Functions and Image Deblurring: Theoretical Analysis and Practice

This paper investigates the role that nonlinear camera response functions (CRFs) have on image deblurring. We present a comprehensive study to analyze the effects of CRFs on motion deblurring. In particular, we show how nonlinear CRFs can cause a spatially invariant blur to behave as a spatially varying blur. We prove that such nonlinearity can cause large errors around edges when directly applying deconvolution to a motion blurred image without CRF correction. These errors are inevitable even with a known point spread function (PSF) and with state-of-the-art regularization-based deconvolution algorithms. In addition, we show how CRFs can adversely affect PSF estimation algorithms in the case of blind deconvolution. To help counter these effects, we introduce two methods to estimate the CRF directly from one or more blurred images when the PSF is known or unknown. Our experimental results on synthetic and real images validate our analysis and demonstrate the robustness and accuracy of our approaches.

Radiometric calibration with illumination change for outdoor scene analysis

The images of an outdoor scene collected over time are valuable in studying the scene appearance variation which can lead to novel applications and help enhance existing methods that were constrained to controlled environments. However, the images do not reflect the true appearance of the scene in many cases due to the radiometric properties of the camera : the radiometric response function and the changing exposure. We introduce a new algorithm to compute the radiometric response function and the exposure of images given a sequence of images of a static outdoor scene where the illumination is changing. We use groups of pixels with constant behaviors towards the illumination change for the response estimation and introduce a sinusoidal lighting variation model representing the daily motion of the sun to compute the exposures.

Radiometric alignment of image sequences

Color values in an image are related to image irradiance by a nonlinear function called radiometric response function. Since this function depends on the aperture and the shutter speed, image intensity of a same object may vary during the acquisition of an image sequence due to auto exposure feature of the camera. While this is desirable to make optimal use of the limited dynamic range of most cameras, this causes problems for a number of applications in computer vision. In this paper we propose a method for estimating the radiometric response function and apply it to radiometrically align images so that the color values are consistent for all images of a sequence. Our approach computes the response function, exposure and white balance changes between images (up to some ambiguity) for a moving camera without any prior knowledge about exposures. We show the performance of our algorithm by estimating the response function from synthetic images and also from real world data, using it to radiometrically align the images.

Revisiting radiometric calibration for color computer vision

We present a study of radiometric calibration and the in-camera imaging process through an extensive analysis of more than 10,000 images from over 30 cameras. The goal is to investigate if image values can be transformed to physically meaningful values and if so, when and how this can be done. From our analysis, we show that the conventional radiometric model fits well for image pixels with low color saturation but begins to degrade as color saturation level increases. This is due to the color mapping process which includes gamut mapping in the in-camera processing that cannot be modeled with conventional methods. To this end, we introduce a new imaging model for radiometric calibration and present an effective calibration scheme that allows us to compensate for the nonlinear color correction to convert non-linear sRGB images to CCD RAW responses.