Miguel Heredia Conde

Subpixel spatial response of PMD pixels

Time-of-Flight cameras based on the Photonic Mixer Device (PMD) emit modulated light and perform a correlation between the light signal reflected by the scene and a reference signal at the pixel level in order to estimate the difference of phase between the emitted and received signals. Well known issues of this technology are the low lateral resolution and the limited and scene-dependent depth accuracy. Superresolution algorithms have been proposed that combine several low resolution depth images to obtain an enhanced high resolution result, or make use of a higher resolution intensity image that can be registered with the depth image. To the best of our knowledge, methods in the related literature seem to ignore systematically the fact that the correlating pixels used for phase-based ToF imagers -e.g., PMD pixels- exhibit a complex structure that might lead to a non-trivial response function in spatial domain. This fact, which is irrelevant when working at pixel level due to the symmetry of pixel structure, becomes of capital importance when working at subpixel level. Ignoring it means assuming a wrong sensing model, which leads to false (or noisy) high resolution recovered images. The aim of this work is to characterize the response function of PMD pixels in spatial domain with micrometric resolution and gather novel and valuable information about the PMD sensing process within the pixel. The results of our experiments clearly confirm our hypothesis about a complex response function and give insights about the origins of noise. Additionally, we obtain a spatial characterization of the crosstalk.

Efficient and Robust Inverse Lighting of a Single Face Image Using Compressive Sensing

We show that the recent theory of Compressive Sensing (CS) can successfully be applied to solve a model-based inverse lighting problem for single face images, even in harsh lighting with multiple light sources, including cast shadows and specularities. It has been shown that an illumination cone can be used to perform realistic inverse lighting. In this work, the cone images are synthetically generated using directional lights and a realistic reflectance of faces. Thereby, the face model is achieved by fitting a 3D Morphable Model to the input image. We apply CS to find the sparsest illumination setup from few random measurements of the RGB input and the cone images. The proposed method significantly reduces the dimensionality through stochastic sampling and a greedy algorithm for the sparse support estimation, yielding low runtimes. The greedy search is designed to handle non-negativity of the light sources and joint-support selection. We show that the proposed method reaches a quality of illumination estimation equal to previous work, while dramatically reducing the number of active light sources. Thorough experimental evaluation shows that stable recovery is achievable for compression rates up to 99%. The method exhibits outstanding robustness to additive noise in the input image.

Adaptive High Dynamic Range for Time-of-Flight Cameras

The limited dynamic range of conventional digital cameras is a well-known problem. A common solution is to apply high-dynamic-range (HDR) techniques over several images acquired using different exposure times. Time-of-flight (ToF) cameras using a photonic mixer device (PMD) are not an exception, since the dynamic range of its dual pixels is also limited. Furthermore, in this case, the saturation of the pixel channels leads to wrong depth measurements. An appropriate solution is the suppression of background illumination (SBI) system designed by the PMD. This system actually extends the dynamic range of the camera by hardware, but it also introduces noise when activated. In this paper, we present an adaptive HDR (AHDR) solution to the problem for the ToF case that overcomes the limited dynamic range of the system, allowing sensing along a theoretically infinite dynamic range with the only limitations of the power of the illumination system and the decay of the SNR with higher distances or lower illumination intensities. Our method is able to detect and segment relevant scene regions responsible for unexpected saturation, i.e., close foreground objects, from the rest of the scene and adjust the exposure times of the acquired images considering them. The results show a reduction in detail losses and a higher SNR in the AHDR raw images, with respect to single acquisitions. This results in a dramatic depth error reduction and effective axial resolution improvement in critical areas, while keeping a high frame rate. In addition, the SBI-related noise is eliminated.

A Compressed Sensing Framework for Accurate and Robust Waveform Reconstruction and Phase Retrieval Using the Photonic Mixer Device

A common way of performing phase-shift-based time-of-flight imaging combines the emission of a continuous-wave (CW) illumination signal with correlation with some reference signals at the detector array. This is the case for the well-known photonic mixer device (PMD), which correlates against displaced versions of the illumination control signal, at known phase shifts, and requires only four correlation values to estimate the phase shift. The main drawback of such approaches is that they require the assumption of nonrealistic hypothesis regarding the sensing process, leading to simple sensing models that, despite allowing fast depth estimation from few acquisitions, often ignore relevant considerations for real operation, leaving the door open for systematic errors that affect the final depth accuracy. Typical examples are ignoring the effect of the illumination devices on the final shape of the illumination signal, supposing a sinusoidal reference signal at pixel level, or not accounting for multipath effects. In this work, we present a novel framework for PMD-based signal acquisition and recovery that exploits the sparsity of CW illumination signals in the frequency domain to provide accurate reconstruction of the illumination waveforms as received by the PMD pixels. Our method is extremely robust to signal distortion and noise, since no assumption is made on the illumination signal, other than being a periodic signal. Our approach ensures that no valuable information is lost during the sensing process and allows, therefore, accurate phase shift estimation in a wider range of operation conditions, getting rid of unrealistic assumptions.

Low-light image enhancement for multiaperture and multitap systems

Intense Poisson noise drastically degrades image quality when only a few or when a single photon hits each pixel. Multiaperture systems are able to provide multiple images of the same scene, which are acquired simultaneously. After registration and cropping, the partial scene information contained in each aperture image should be the same, while the noise will be different in each one. A similar case arises in multitap systems, which are widely used in Time-of-Flight imaging (ToF), where several integration channels per pixel exist and where several sequential acquisitions are needed to generate a depth image. In this case, raw images might be different from each other, but still, since they are images of the same scene, information redundancy can be exploited to filter out the noise. In this work, we propose two different ways of joint processing of low-light multiaperture images. One of them is an extension of bilateral filtering to the multiaperture case, while the other relies on the compressive sensing theory and aims to recover a noiseless image from fewer measurements than the total number of pixels in the original noisy images. Experimental results show that both methods exhibit very close performance, which is much higher than those of previous methods. Additionally, we show that bilateral filtering can also be applied to the raw images of multitap ToF systems, leading to a significant error reduction in the final depth image.

Compressive Sensing for the Photonic Mixer Device

At the light of the fundamentals of Compressive Sensing (CS) presented in Chapter 3, it is clear that the phase-shift-based ToF imaging systems described in Chapter 2 offer an appropriate application area of CS. The signals to be sensed are typically not random, but follow a specific structure, which allows them to be sparsely represented in an appropriate dictionary and, therefore, eventually recovered from few measurements. In this chapter we study the most appealing ways of applying CS to PMD-based ToF imaging systems, with the main goal of overcoming the current limits of the technology, introduced in Section 2.4, mostly the poor lateral resolution, derived from the low number of pixels, and the low depth accuracy.

Crosstalk characterization of PMD pixels using the spatial response function at subpixel level

Time-of-Flight cameras have become one of the most widely-spread low-cost 3D-sensing devices. Most of them do not actually measure the time the light needs to hit an object and come back to the camera, but the difference of phase with respect to a reference signal. This requires special pixels with complex spatial structure, such as PMD pixels, able to sample the cross-correlation function between the incoming signal, reflected by the scene, and the reference signal. The complex structure, together with the presence of in-pixel electronics and the need for a compact readout circuitry for both pixel channels, suggests that systematic crosstalk effects will come up in this kind of devices. For the first time, we take profit of recent results on subpixel spatial responses of PMD pixels to detect and characterize crosstalk occurrences. Well-defined crosstalk patterns have been identified and quantitatively characterized through integration of the inter-pixel spatial response over each sensitive area. We cast the crosstalk problem into an image convolution and provide deconvolution kernels for cleaning PMD raw images from crosstalk. Experiments on real PMD raw images show that our results can be used to undo the lowpass filtering caused by crosstalk in high contrast image areas. The application of our kernels to undo crosstalk effects leads to reductions of the depth RMSE up to 50% in critical areas.

Simultaneous Multichannel Waveform Recovery of Illumination Signals Using Compressed Sensing

A critical element of any time-of-flight (ToF) 3D imaging system is the illumination. Most commercial solutions are restricted to short range indoor operation and use simple illumination setups of single or few LEDs, grouped together. Recent developments toward medium and long range ToF imaging, ready for outdoor operation, bring the need for powerful illumination setups, constituted by many emitters, which might be grouped in distributed modules. Provided that the depth accuracy of ToF cameras strongly depends on the quality of the illumination waveform, assuring that a complex illumination system is providing a homogeneous in-phase wavefront is of capital importance to minimize systematic inaccuracies. In this letter, we present a novel framework for multichannel simultaneous testing of illumination waveforms, which is able to recover the waveform of the incident light on each pixel of a ToF camera, exploiting the sparsity of typical continuous wave illumination signals in frequency domain.