J. P. Lüke

3D imaging and wavefront sensing with a plenoptic objective

Plenoptic cameras have been developed over the last years as a passive method for 3d scanning. Several superresolution algorithms have been proposed in order to increase the resolution decrease associated with lightfield acquisition with a microlenses array. A number of multiview stereo algorithms have also been applied in order to extract depth information from plenoptic frames. Real time systems have been implemented using specialized hardware as Graphical Processing Units (GPUs) and Field Programmable Gates Arrays (FPGAs). In this paper, we will present our own implementations related with the aforementioned aspects but also two new developments consisting of a portable plenoptic objective to transform every conventional 2d camera in a 3D CAFADIS plenoptic camera, and the novel use of a plenoptic camera as a wavefront phase sensor for adaptive optics (OA). The terrestrial atmosphere degrades the telescope images due to the diffraction index changes associated with the turbulence. These changes require a high speed processing that justify the use of GPUs and FPGAs. Na artificial Laser Guide Stars (Na-LGS, 90km high) must be used to obtain the reference wavefront phase and the Optical Transfer Function of the system, but they are affected by defocus because of the finite distance to the telescope. Using the telescope as a plenoptic camera allows us to correct the defocus and to recover the wavefront phase tomographically. These advances significantly increase the versatility of the plenoptic camera, and provides a new contribution to relate the wave optics and computer vision fields, as many authors claim.

Depth From Light Fields Analyzing 4D Local Structure

In this paper, we develop a local method to obtain depths from the 4D light field. In contrast to previous local depth from light field methods based on EPIs, e.g., 2D slices of the light field, the proposed method takes into account the 4D nature of the light field and uses its four dimensions. Furthermore, our technique adapts well to parallel hardware. The performance of the method is tested against a publicly available benchmark dataset and compared with other algorithms that previously have been tested with the same benchmark. Results show that the proposed method can achieve competitive results in reasonable time.

Design of Belief Propagation Based on FPGA for the Multistereo CAFADIS Camera

In this paper we describe a fast, specialized hardware implementation of the belief propagation algorithm for the CAFADIS camera, a new plenoptic sensor patented by the University of La Laguna. This camera captures the lightfield of the scene and can be used to find out at which depth each pixel is in focus. The algorithm has been designed for FPGA devices using VHDL. We propose a parallel and pipeline architecture to implement the algorithm without external memory. Although the BRAM resources of the device increase considerably, we can maintain real-time restrictions by using extremely high-performance signal processing capability through parallelism and by accessing several memories simultaneously. The quantifying results with 16 bit precision have shown that performances are really close to the original Matlab programmed algorithm.

Atmospherical wavefront phases using the plenoptic sensor (real data)

Plenoptic cameras have been developed the last years as a passive method for 3d scanning, allowing focal stack capture from a single shot. But data recorded by this kind of sensors can also be used to extract the wavefront phases associated to the atmospheric turbulence in an astronomical observation. The terrestrial atmosphere degrades the telescope images due to the diffraction index changes associated to the turbulence. Na artificial Laser Guide Stars (Na-LGS, 90km high) must be used to obtain the reference wavefront phase and the Optical Transfer Function of the system, but they are affected by defocus because of the finite distance to the telescope. Using the telescope as a plenoptic camera allows us to correct the defocus and to recover the wavefront phase tomographically, taking advantage of the two principal characteristics of the plenoptic sensors at the same time: 3D scanning and wavefront sensing. Then, the plenoptic sensors can be studied and used as an alternative wavefront sensor for Adaptive Optics, particularly relevant when Extremely Large Telescopes projects are being undertaken. In this paper, we will present the first observational wavefront phases extracted from real astronomical observations, using punctual and extended objects, and we show that the restored wavefronts match the Kolmogorov atmospheric turbulence.

Fast approximate 4-D/3-D discrete radon transform for lightfield refocusing

We develop a new algorithm that extends the bidimensional fast digital radon transform from Gotz and Druckmuller (1996) to digitally simulate the refocusing of a 4-D lightfield into a 3-D volume of photographic planes as previously done by Ng et al. (2005) but with the minimum number of operations. This new algorithm does not require multiplications just sums and its computational complexity is O(N 4 ) to achieve a volume consisting of 2 N photographic planes focused at different depths from a N 4 plenoptic image. This reduced complexity allows for the acquisition and processing of a plenoptic sequence with the purpose of estimating 3-D shape at video rate. Examples are given of implementations on GPU and CPU platforms. Finally, a modified version of the algorithm to deal with domains of sizes different than a power of two is proposed.

Design and Laboratory Results of a Plenoptic Objective: From 2D to 3D With a Standard Camera

The plenoptic camera was originally created to allow the capture of the light field, a four-variable volume representation of all rays and their directions, which allows the creation by synthesis of an image of the observed object. This method has several advantages with regard to 3D capture systems based on stereo cameras since it does not need frame synchronization or geometric and color calibration. It also has many applications, from 3DTV to medical imaging. A plenoptic camera uses a microlens array to measure the radiance and direction of all the light rays in a scene. The array is placed at a distance from the principal lens, which is conjugated to the distance where the scene is situated, and the sensor is at the focal plane of the microlenses. We have designed a plenoptic objective that incorporates a microlens array and a relay system that reimages the microlens plane. This novel approach has proven successful. Placing it on a camera, the plenoptic objective creates a virtual microlens plane in front of the camera CCD, allowing it to capture the light field of the scene. In this paper we present the experimental results showing that depth information is perfectly captured when using an external plenoptic objective. Using this objective transforms any camera into a 3D sensor, opening up a wide range of applications from microscopy to astronomy .

Fast approximate 4D:3D discrete radon transform, from light field to focal stack with O(N4) sums

In this work we develop a new algorithm, that extends the bidimensional Fast Digital Radon transform from Götz and Druckmüller (1996), to digitally simulate the refocusing of a 4D light field into a 3D volume of photographic planes, as previously done by Ren Ng et al. (2005), but with the minimum number of operations. This new algorithm does not require multiplications, just sums, and its computational complexity is O(N4) to achieve a volume consisting of 2N photographic planes focused at different depths, from a N4 plenoptic image. This reduced complexity allows for the acquisition and processing of a plenoptic sequence with the purpose of estimating 3D shape at video rate. Examples are given of implementations on GPU and CPU platforms. Finally, a modified version of the algorithm to deal with domains of sizes different than power of two, is proposed.

Error analysis of depth estimations based on orientation detection in EPI-representations of 4D light fields

In the last years, interest on depth extraction from 4D light fields has increased. These techniques rely on the detection of slopes of planar structures in the light field function. These structures are also found as linear structures in epipolar image representations of the light field. Since EPI-representations are 2D signals, known orientation detection methods can be applied. This work shows that orientation estimation suffers from two types of errors: Systematic errors and random errors. A theoretical expression for the standard deviation of random errors has been formulated and verified.

On the application of the plenoptic camera to mobile phones

The plenoptic camera was originally created to allow the capture of the Light Field, a four-variable volume representation of all rays and their directions, that allows the creation by synthesis of a 3D image of the observed object. This method has several advantages with regard to 3D capture systems based on stereo cameras, since it does note need frame synchronization or geometric and color calibration. And it has many applications, from 3DTV to medical imaging. A plenoptic camera uses a microlens array to measure the radiance and direction of all the light rays in a scene. The array is placed at the focal plane of the objective lens, and the sensor is at the focal plane of the microlenses. In this paper we study the application of our super resolution algorithm to mobile phones cameras. With a commercial camera, it is already possible to obtain images of good resolution and enough number of refocused planes, just placing a microlens array in front of the detector.

Teaching methodology of digital electronic systems subject in the new degree of computers

In this paper we describe changes in a subject of digital logic design to fit the new plan of Bologna. The subject has a significant increase in hours of laboratory and students compared to previous years. We present the results of a student satisfaction survey and a comparison of the success of overcoming the remaining subject compared to past years.