Bert Geelen

A compact snapshot multispectral imager with a monolithically integrated per-pixel filter mosaic

The adoption of spectral imaging by industry has so far been limited due to the lack of high speed, low cost and compact spectral cameras. Moreover most state-of-the-art spectral cameras utilize some form of spatial or spectral scanning during acquisition, making them ill-suited for analyzing dynamic scenes containing movement. This paper introduces a novel snapshot multispectral imager concept based on optical filters monolithically integrated on top of a standard CMOS image sensor. It overcomes the problems mentioned for scanning applications by snapshot acquisition, where an entire multispectral data cube is sensed at one discrete point in time. This is enabled by depositing interference filters per pixel directly on a CMOS image sensor, extending the traditional Bayer color imaging concept to multi- or hyperspectral imaging without a need for dedicated fore-optics. The monolithic deposition leads to a high degree of design flexibility. This enables systems ranging from application-specific, high spatial resolution cameras with 1 to 4 spectral filters, to hyperspectral snapshot cameras at medium spatial resolutions and filters laid out in cells of 4x4 to 6x6 or more. Through the use of monolithically integrated optical filters it further retains the qualities of compactness, low cost and high acquisition speed, differentiating it from other snapshot spectral cameras.

A snapshot multispectral imager with integrated tiled filters and optical duplication

Although the potential of spectral imaging has been demonstrated in research environments, its adoption by industry has so far been limited due to the lack of high speed, low cost and compact spectral cameras. We have previously presented work to overcome this limitation by monolithically integrating optical interference filters on top of standard CMOS image sensors for high resolution pushbroom hyperspectral cameras. These cameras require a scanning of the scene and therefore introduce operator complexity due to the need for synchronization and alignment of the scanning to the camera. This typically leads to problems with motion blur, reduced SNR in high speed applications and detection latency and overall restricts the types of applications that can use this system. This paper introduces a novel snapshot multispectral imager concept based on optical filters monolithically integrated on top of a standard CMOS image sensor. By using monolithic integration for the dedicated, high quality spectral filters at its core, it enables the use of mass-produced fore-optics, reducing the total system cost. It overcomes the problems mentioned for scanning applications by snapshot acquisition, where an entire multispectral data cube is sensed at one discrete point in time. This is achieved by applying a novel, tiled filter layout and an optical sub-system which simultaneously duplicates the scene onto each filter tile. Through the use of monolithically integrated optical filters it retains the qualities of compactness, low cost and high acquisition speed, differentiating it from other snapshot spectral cameras based on heterogeneously integrated custom optics. Moreover, thanks to a simple cube assembly process, it enables real-time, low-latency operation. Our prototype camera can acquire multispectral image cubes of 256x256 pixels over 32 bands in the spectral range of 600-1000nm at a speed of about 30 cubes per second at daylight conditions up to 340 cubes per second at higher illumination levels as typically used in machine vision applications.

A tiny VIS-NIR snapshot multispectral camera

Spectral imaging can reveal a lot of hidden details about the world around us, but is currently confined to laboratory environments due to the need for complex, costly and bulky cameras. Imec has developed a unique spectral sensor concept in which the spectral unit is monolithically integrated on top of a standard CMOS image sensor at wafer level, hence enabling the design of compact, low cost and high acquisition speed spectral cameras with a high design flexibility. This flexibility has previously been demonstrated by imec in the form of three spectral camera architectures: firstly a high spatial and spectral resolution scanning camera, secondly a multichannel snapshot multispectral camera and thirdly a per-pixel mosaic snapshot spectral camera. These snapshot spectral cameras sense an entire multispectral data cube at one discrete point in time, extending the domain of spectral imaging towards dynamic, video-rate applications. This paper describes the integration of our per-pixel mosaic snapshot spectral sensors inside a tiny, portable and extremely user-friendly camera. Our prototype demonstrator cameras can acquire multispectral image cubes, either of 272x512 pixels over 16 bands in the VIS (470-620nm) or of 217x409 pixels over 25 bands in the VNIR (600-900nm) at 170 cubes per second for normal machine vision illumination levels. The cameras themselves are extremely compact based on Ximea xiQ cameras, measuring only 26x26x30mm, and can be operated from a laptop-based USB3 connection, making them easily deployable in very diverse environments.

Low-complexity image processing for a high-throughput low-latency snapshot multispectral imager with integrated tiled filters

Traditional spectral imaging cameras typically operate as pushbroom cameras by scanning a scene. This approach makes such cameras well-suited for high spatial and spectral resolution scanning applications, such as remote sensing and machine vision, but ill-suited for 2D scenes with free movement. This limitation can be overcome by single frame, multispectral (here called snapshot) acquisition, where an entire three-dimensional multispectral data cube is sensed at one discrete point in time and multiplexed on a 2D sensor. Our snapshot multispectral imager is based on optical filters monolithically integrated on CMOS image sensors with large layout flexibility. Using this flexibility, the filters are positioned on the sensor in a tiled layout, allowing trade-offs between spatial and spectral resolution. At system-level, the filter layout is complemented by an optical sub-system which duplicates the scene onto each filter tile. This optical sub-system and the tiled filter layout lead to a simple mapping of 3D spectral cube data on the sensor, facilitating simple cube assembly. Therefore, the required image processing consists of simple and highly parallelizable algorithms for reflectance and cube assembly, enabling real-time acquisition of dynamic 2D scenes at low latencies. Moreover, through the use of monolithically integrated optical filters the multispectral imager achieves the qualities of compactness, low cost and high acquisition speed, further differentiating it from other snapshot spectral cameras. Our prototype camera can acquire multispectral image cubes of 256x256 pixels over 32 bands in the spectral range of 600-1000nm at 340 cubes per second for normal illumination levels.

Generalized inpainting method for hyperspectral image acquisition

A recently designed hyperspectral imaging device enables multiplexed acquisition of an entire data volume in a single snapshot thanks to monolithically-integrated spectral filters. Such an agile imaging technique comes at the cost of a reduced spatial resolution and the need for a demosaicing procedure on its interleaved data. In this work, we address both issues and propose an approach inspired by recent developments in compressed sensing and analysis sparse models. We formulate our superresolution and demosaicing task as a 3-D generalized inpainting problem. Interestingly, the target spatial resolution can be adjusted for mitigating the compression level of our sensing. The reconstruction procedure uses a fast greedy method called Pseudo-inverse IHT. We also show on simulations that a random arrangement of the spectral filters on the sensor is preferable to regular mosaic layout as it improves the quality of the reconstruction. The efficiency of our technique is demonstrated through numerical experiments on both synthetic and real data as acquired by the snapshot imager.

A novel CMOS-compatible, monolithically integrated line-scan hyperspectral imager covering the VIS-NIR range

Imec has developed a process for the monolithic integration of optical filters on top of CMOS image sensors, leading to compact, cost-efficient and faster hyperspectral cameras. Different prototype sensors are available, most notably a 600- 1000 nm line-scan imager, and two mosaic sensors: a 4x4 VIS (470-620 nm range) and a 5x5 VNIR (600-1000 nm). In response to the users’ demand for a single sensor able to cover both the VIS and NIR ranges, further developments have been made to enable more demanding applications. As a result, this paper presents the latest addition to imec’s family of monolithically-integrated hyperspectral sensors: a line scan sensor covering the range 470-900 nm. This new prototype sensor can acquire hyperspectral image cubes of 2048 pixels over 192 bands (128 bands for the 600- 900 nm range, and 64 bands for the 470-620 nm range) at 340 cubes per second for normal machine vision illumination levels.

Exploration of system-level trade-offs for application mapping in multiprocessor system-on-chips

In order to meet heavy constraints for power, performance and flexibility, applications on future hand-held devices will be executed on architectures with multiple processors. However, this will only succeed if the application is mapped using an efficient, coarse-grained parallelization. The final performance of the parallelexecuted applications heavily depends on the partitioning, the architecture characteristics, the memory hierarchy, ... Since it is very hard for a designer to oversee the effect of these interactions manually, tools should be developed. The current PhD explores these issues based on real-life examples of wavelet still image, video and volumetric coding.

System-level analysis and design for RGB-NIR CMOS camera

This paper presents system-level analysis of a sensor capable of simultaneously acquiring both standard absorption based RGB color channels (400-700nm, ~75nm FWHM), as well as an additional NIR channel (central wavelength: ~808 nm, FWHM: ~30nm collimated light). Parallel acquisition of RGB and NIR info on the same CMOS image sensor is enabled by monolithic pixel-level integration of both a NIR pass thin film filter and NIR blocking filters for the RGB channels. This overcomes the need for a standard camera-level NIR blocking filter to remove the NIR leakage present in standard RGB absorption filters from ~700-1000nm. Such a camera-level NIR blocking filter would inhibit the acquisition of the NIR channel on the same sensor. Thin film filters do not operate in isolation. Rather, their performance is influenced by the system context in which they operate. The spectral distribution of light arriving at the photo diode is shaped a.o. by the illumination spectral profile, optical component transmission characteristics and sensor quantum efficiency. For example, knowledge of a low quantum efficiency (QE) of the CMOS image sensor above 800nm may reduce the filter’s blocking requirements and simplify the filter structure. Similarly, knowledge of the incoming light angularity as set by the objective lens’ F/# and exit pupil location may be taken into account during the thin film’s optimization. This paper demonstrates how knowledge of the application context can facilitate filter design and relax design trade-offs and presents experimental results.

RGB-NIR active gated imaging

This paper presents multispectral active gated imaging in relation to the transportation and security fields. Active gated imaging is based on a fast gated camera and pulsed illuminator, synchronized in the time domain to provide range based images. We have developed a multispectral pattern deposited on a gated CMOS Image Sensor (CIS) with a pulsed Near Infrared VCSEL module. This paper will cover the component-level description of the multispectral gated CIS including the camera and illuminator units. Furthermore, the design considerations and characterization results of the spectral filters are presented together with a newly developed image processing method.