Michael Schöberl

Photometric limits for digital camera systems

Image sensors for digital cameras are built with ever decreasing pixel sizes. The size of the pixels seems to be limited by technology only. However, there is also a hard theoretical limit for classical video camera systems: During a certain exposure time only a certain number of photons will reach the sensor. The resulting shot noise thus limits the signal-to-noise ratio. In this letter we show that current sensors are already surprisingly close to this limit.

Increasing imaging resolution by covering your sensor

Up to now, an increase in camera resolution required image sensors with more and more pixels. However, acquisition systems are limited in their pixels per second throughput given as power and complexity constraints. Simply capturing more pixels in a given system is often not possible. We propose a new non-regular imaging architecture that samples only few pixels and reconstructs a high resolution image afterwards. Our sampling is optimized to provide non-regular spatial sampling from a sensor with regular readout circuits. An existing slow image acquisition system can then be used to capture the data. The image reconstruction is performed with a local sparsity-based approach. The result is a high resolution image that requires a much smaller effort during acquisition.

High dynamic range video by spatially non-regular optical filtering

We present a new method for capturing high dynamic range video (HDRV). Our method is based on spatially varying exposures, where individual pixels are covered with filters for different optical attenuation. For preventing the loss in resolution we use a new non-regular arrangement of the attenuation pattern. Subsequent image reconstruction based on the sparsity assumption allows the reconstruction of natural images with high detail.

Diffraction and photometric limits in today's miniature digital camera systems

Image sensors for digital cameras are built with ever decreasing pixel sizes. The size of the pixels seems to be limited by technology only. However, there are also hard theoretical limits for classical miniature camera systems: During a certain exposure time only a certain number of photons will reach the sensor. The resulting shot noise thus limits the signal-to-noise ratio. On the other hand, diffraction sets another limit for image resolution in case that there is enough brightness in the scene. In this work we show that current sensors are already surprisingly close to these limits.

Dimensioning of optical birefringent anti-alias filters for digital cameras

A digital camera samples the continuous real world. As with any sampling process, questions of aliasing for certain sampling frequencies and the prevention thereof arise. In this paper we will discuss the spatial domain sampling and prevention of aliasing in digital cameras. We focus on the widely used birefringent anti alias filters that are often called optical low pass filters (OLPF). We show 2D models for all contributions to spatial domain sampling and derive optimum filter parameters for minimum aliasing and best possible image sharpness. Compared to previously used selection rules, we can show that the optimum selection of filter parameters can easily deliver more sharpness and reduce aliasing by a factor of 2. The simulated results are finally confirmed in real world experiments.

Building a high dynamic range video sensor with spatially nonregular optical filtering

Although we observe a steady progress in the development of High Dynamic Range Video (HDRV) technology, current image sensors are still lacking achievable dynamic range for high image quality applications. We propose a new imaging principle that is based on a spatial variation of optical Neutral Density (ND) filters on top of some pixels. In existing work, this method has been used to trade spatial resolution for an increase in dynamic range. We improve this approach by a non-regular placement of these filters. The non-regular sampling is an important step as any sub-sampling with regular patterns leads to aliasing. The non-regular patterns however preserve just a single dominant spatial frequency and enable an image reconstruction without aliasing. In combination with a new image reconstruction approach, we are able to recover image details at high resolution. The iterative reconstruction is based on the assumption that natural images can be represented with few coefficients in the Fourier domain. As typical natural images can be classified as near-sparse, the method enables the reconstruction of images of high objective and visual quality. In extension to theory and simulation of this method we want to present details on a practical implementation of our method. While building a demonstration system we encountered many challenges. This includes effects like crosstalk, aspects like sensor selection and mask fabrication as well as mounting of the masks.

Modeling of image shutters and motion blur in analog and digital camera systems

For motion imaging the perceived smoothness of a sequence highly depends on motion blur. The exposure for each frame is started and ended with a shutter mechanism. There are different implementations and some of them have non-ideal behavior which can introduce artifacts. In this paper an overview of real-world shutter implementations both for analog and digital camera systems is shown. We develop a general description that models all types of shutters and their imperfections. Specific models for common shutter types are presented. Measurements are used to estimate unknown parameters. The modeled shutters are finally used for a virtual camera simulation. Typical artifacts can be simulated and directly compared for different shutter types and parameters. This powerful tool is useful for the construction of camera systems and allows design decisions to be directly compared before building the camera system.

Sparsity-based defect pixel compensation for arbitrary camera raw images

In high quality imaging even tiny distortions as small as a single pixel are visible and can not be accepted. Although the production quality of CMOS image sensors is very high, for reasonable yields we still need to accept some defect pixels and clusters of defects in large image sensors. In this paper we will compare compensation algorithms for raw image sensor data. We propose a new approach based on the sparsity assumption that outperforms existing defect compensation algorithms. Furthermore, our proposed interpolation algorithm is universal and not at all adapted to Bayer pattern images. It can directly be applied to any regular color filter pattern or gray scale image. Our examples show, that image sensors with large clusters of defects can still be used for the generation of high quality images.

Fixed pattern noise column drift compensation (CDC) for digital moving picture cameras

In CMOS image sensors tiny semiconductor variations cause a distortion of the image known as fixed pattern noise (FPN). With a good FPN compensation CMOS sensors can deliver very good image quality. Compensation algorithms differ in complexity, maximum frame rate, and compensation quality with thermal drift. This paper analyzes existing approaches to offset FPN compensation and their drawbacks. We also show a new compensation algorithm that offers fine grained per-pixel compensation and is able to compensate temperature variations while still operating at the maximum sensor frame rate. This enables the construction of motion picture cameras without active cooling or even temperature measurements while still delivering a high image quality at high frame rates.

Mixed-resolution view synthesis using non-local means refined image merging

Synthesizing novel views from originally available camera perspectives via depth maps is a key issue in the 3D video domain. Up to now, several high-resolution cameras are needed to obtain high-quality intermediate synthesized views. One possibility to reduce costs with regard to the used camera array is to replace some cameras by low-resolution cameras, which are cheaper on the one hand, but provide a much poorer image quality on the other hand. Unfortunately, some of the information inside the desired intermediate view may only be available in the low-resolution reference. Thus, the image quality of the low-resolution reference has a big influence on the visual quality of the synthesized view. This paper proposes a postprocessing step for the synthesized view, based on the non-local means algorithm. Thereby, all areas inserted from the low-resolution reference get efficiently adapted to their high-resolution environment. It is shown, that the non-local means refined image merging leads to a PSNR gain of up to 0.90 dB compared to an unrefined mixed-resolution setup. The approach can be easily extended to a hole-filling algorithm and yields a PSNR gain of up to 0.81 dB for hole areas compared to a reference hole-filling algorithm. The subjective image quality also increases convincingly in both applications.