Yusuke Monno

Residual interpolation for color image demosaicking

A color difference interpolation technique is widely used for color image demosaicking. In this paper, we propose residual interpolation as an alternative to the color difference interpolation, where the residual is a difference between an observed and a tentatively estimated pixel value. We incorporate the proposed residual interpolation into the gradient based threshold free (GBTF) algorithm, which is one of current state-of-the-art demosaicking algorithms. Experimental results demonstrate that our proposed demosaicking algorithm using the residual interpolation can give state-of-the-art performance for the 30 images of Kodak and IMAX datasets.

Multispectral demosaicking using adaptive kernel upsampling

Multispectral demosaicking, which estimates full multispectral images from raw data observed using a single image sensor with a color filter array (CFA), is a challenging task because each spectral component is severely undersampled. In this paper, we propose a novel multispectral demosaicking algorithm. We extend existing upsampling algorithms to adaptive kernel upsampling algorithms using an adaptive kernel as a spatial weight and apply them to multispectral demosaicking. We also propose a new CFA and a direct adaptive kernel estimation from the raw data of the proposed CFA. Experimental results with real multispectral images demonstrate the effectiveness of the proposed algorithm.

Minimized-Laplacian residual interpolation for color image demosaicking

A color difference interpolation technique is widely used for color image demosaicking. In this paper, we propose a minimized-laplacian residual interpolation (MLRI) as an alternative to the color difference interpolation, where the residuals are differences between observed and tentatively estimated pixel values. In the MLRI, we estimate the tentative pixel values by minimizing the Laplacian energies of the residuals. This residual image transfor- mation allows us to interpolate more easily than the standard color difference transformation. We incorporate the proposed MLRI into the gradient based threshold free (GBTF) algorithm, which is one of current state-of- the-art demosaicking algorithms. Experimental results demonstrate that our proposed demosaicking algorithm can outperform the state-of-the-art algorithms for the 30 images of the IMAX and the Kodak datasets.

Multispectral demosaicking using guided filter

Multispectral imaging is highly demanded for precise color reproduction and for various computer vision applications. Multispectral imaging with a multispectral color filter array (MCFA), which can be considered as a multispectral extension of commonly used consumer RGB cameras, could be a simple, low-cost, and practical system. A challenge of the multispectral imaging with the MCFA is multispectral demosaicking because each spectral component of the MCFA is severely undersampled. In this paper, we propose a novel multispectral demosaicking algorithm using a guided filter. The guided filter is recently proposed as an excellent structurepreserving filter. The guided filter requires so-called a guide image. A main issue of the guided filter is how to obtain an effective guide image. In our proposed algorithm, we generate the guide image from the most densely sampled spectral component in the MCFA. Then, ohter spectral components are interpolated by the guided filter. Experimental results demonstrate that our proposed algorithm outperforms other existing demosaicking algorithms both visually and quantitatively.

Beyond Color Difference: Residual Interpolation for Color Image Demosaicking

In this paper, we propose residual interpolation (RI) as an alternative to color difference interpolation, which is a widely accepted technique for color image demosaicking. Our proposed RI performs the interpolation in a residual domain, where the residuals are differences between observed and tentatively estimated pixel values. Our hypothesis for the RI is that if image interpolation is performed in a domain with a smaller Laplacian energy, its accuracy is improved. Based on the hypothesis, we estimate the tentative pixel values to minimize the Laplacian energy of the residuals. We incorporate the RI into the gradient-based threshold free algorithm, which is one of the state-of-the-art Bayer demosaicking algorithms. Experimental results demonstrate that our proposed demosaicking algorithm using the RI surpasses the state-of-the-art algorithms for the Kodak, the IMAX, and the beyond Kodak data sets.

Optimal spectral sensitivity functions for a single-camera one-shot multispectral imaging system

Multispectral imaging is highly demanded for precise color reproduction and for various computer vision applications. Recently, a single-camera one-shot multispectral imaging (SCOS) system that uses a single image sensor equipped with a multispectral filter array (MSFA) has been proposed. In this paper, we develop optimal spectral sensitivity functions (SSFs) for the SCOS system, in which multispectral image quality depends strongly on the performance of multispectral demosaicking. First, we propose a simple optimization algorithm that can incorporate a high-performance multispectral demosaicking algorithm. Then, we experimentally demonstrate that the optimized SSFs by our proposed algorithm improve the performance of spectral reflectance estimation and the accuracy of color reproduction.

Adaptive residual interpolation for color image demosaicking

Color image demosaicking is an essential image processing operation for acquiring high-quality color images. Recently, demosaicking algorithms using residual interpolation (RI), which performs the interpolation in a residual domain, have been proposed. An iterative framework has also been introduced into the RI and shown state-of-the-art performance. In this paper, we propose a novel demosaicking algorithm using adaptive residual interpolation (ARI), which adaptively selects a suitable iteration number and combines two different types of RI algorithms at each pixel. Experimental results demonstrate that our demosaicking algorithm can achieve a clear improvement in comparison with existing algorithms.

Multispectral demosaicking with novel guide image generation and residual interpolation

A one-shot multispectral imaging system using a multispectral filter array (MSFA) provides a practical solution for compact, low-cost, and real-time multispectral imaging. However, multispectral demosaicking is a challenging problem because each spectral band is significantly undersampled in the MSFA. In this paper, we propose a novel demosaicking algorithm for the MSFA proposed in [1, 2]. Main contributions of this paper are (i) we utilize multispectral correlations for generating a guide image, which is effectively used for interpolation preserving image structures, and (ii) we effectively use residual interpolation (RI) [3] for generating the guide image and interpolating each spectral band. Experimental results demonstrate that our proposed algorithm significantly outperforms existing state-of-the-art algorithms.

Single-Sensor RGB and NIR Image Acquisition: Toward Optimal Performance by Taking Account of CFA Pattern, Demosaicking, and Color Correction.

In recent years, many applications using a pair of RGB and near-infrared (NIR) images have been proposed in computer vision and image processing communities. Thanks to recent progress of image sensor technology, it is also becoming possible to manufacture an image sensor with a novel spectral filter array, which has RGB plus NIR pixels for one-shot acquisition of the RGB and the NIR images. In such a novel filter array, half of the G pixels in the standard Bayer color filter array (CFA) are typically replaced with the NIR pixels. However, its performance has not fully been investigated in the pipeline of single-sensor RGB and NIR image acquisition. In this paper, we present an imaging pipeline of the single-sensor RGB and NIR image acquisition and investigate its optimal performance by taking account of the filter array pattern, demosaicking and color correction. We also propose two types of filter array patterns and demosaicking algorithms for improving the quality of acquired RGB and NIR images. Based on the imaging pipeline we present, the performance of different filter array patterns and demosaicking algorithms is evaluated. In experimental results, we demonstrate that our proposed filter array patterns and demosaicking algorithms outperform the existing ones. Introduction In recent years, many applications using a pair of RGB and near-infrared (NIR) images have been proposed in computer vision and image processing communities such as image enhancement [1, 2], image fusion [3, 4], dehazing [5, 6], denoising [7, 8], and shadow detection [9]. However, the acquisition of the pair of RGB and NIR images is still a challenging task because existing acquisition systems typically require multiple cameras [1] or multiple shots [9], where one is required for RGB and the other is required for NIR. In current compact and low-cost digital cameras, singlesenor color image acquisition with the Bayer color filter array (CFA) [10], as shown in Fig. 1 (a)-(c), is well established [11]. To simultaneously acquire the RGB and the NIR images, many existing works extend the idea of using the CFA for single-sensor RGB and NIR image acquisition [12–18]. Thanks to recent progress of image sensor technology, it is also becoming possible to manufacture an image sensor with a novel filter array, which has RGB plus NIR pixels [14–18]. This sensor can provide us with a practical solution for one-shot acquisition of the RGB and the NIR images without increased size and cost from current color digital cameras. Hereafter, we call such a filter array “RGB-NIR filter 0 5 10 15 20 25 30 35 40 400 500 600 700 800 90

Direct spatio-spectral datacube reconstruction from raw data using a spatially adaptive spatio-spectral basis

Spectral reflectance is an inherent property of objects that is useful for many computer vision tasks. The spectral reflectance of a scene can be described as a spatio-spectral (SS) datacube, in which each value represents the reflectance at a spatial location and a wavelength. In this paper, we propose a novel method that reconstructs the SS datacube from raw data obtained by an image sensor equipped with a multispectral filter array. In our proposed method, we describe the SS datacube as a linear combination of spatially adaptive SS basis vectors. In a previous method, spatially invariant SS basis vectors are used for describing the SS datacube. In contrast, we adaptively generate the SS basis vectors for each spatial location. Then, we reconstruct the SS datacube by estimating the linear coefficients of the spatially adaptive SS basis vectors from the raw data. Experimental results demonstrate that our proposed method can accurately reconstruct the SS datacube compared with the method using spatially invariant SS basis vectors.