Tao Yue

Computational Snapshot Multispectral Cameras: Toward dynamic capture of the spectral world

Multispectral cameras collect image data with a greater number of spectral channels than traditional trichromatic sensors, thus providing spectral information at a higher level of detail. Such data are useful in various fields, such as remote sensing, materials science, biophotonics, and environmental monitoring. The massive scale of multispectral data-at high resolutions in the spectral, spatial, and temporal dimensions-has long presented a major challenge in spectrometer design. With recent developments in sampling theory, this problem has become more manageable through use of undersampling and constrained reconstruction techniques. This article presents an overview of these state-of-the-art multispectral acquisition systems, with a particular focus on snapshot multispectral capture, from a signal processing perspective. We propose that undersampling-based multispectral cameras can be understood and compared by examining the efficiency of their sampling schemes, which we formulate as the spectral sensing coherence information between their sensing matrices and spectrum-specific bases learned from a large-scale multispectral image database. We analyze existing snapshot multispectral cameras in this manner, and additionally discuss their optical performance in terms of light throughput and system complexity.

Patch-primitive driven compressive ghost imaging

Ghost imaging has rapidly developed for about two decades and attracted wide attention from different research fields. However, the practical applications of ghost imaging are still largely limited, by its low reconstruction quality and large required measurements. Inspired by the fact that the natural image patches usually exhibit simple structures, and these structures share common primitives, we propose a patch-primitive driven reconstruction approach to raise the quality of ghost imaging. Specifically, we resort to a statistical learning strategy by representing each image patch with sparse coefficients upon an over-complete dictionary. The dictionary is composed of various primitives learned from a large number of image patches from a natural image database. By introducing a linear mapping between non-overlapping image patches and the whole image, we incorporate the above local prior into the convex optimization framework of compressive ghost imaging. Experiments demonstrate that our method could obtain better reconstruction from the same amount of measurements, and thus reduce the number of requisite measurements for achieving satisfying imaging quality.

The role of prior in image based 3D modeling: a survey

The prior knowledge is the significant supplement to image-based 3D modeling algorithms for refining the fragile consistency-based stereo. In this paper, we review the image-based 3D modeling problem according to prior categories, i.e., classical priors and specific priors. The classical priors including smoothness, silhouette and illumination are well studied for improving the accuracy and robustness of the 3D reconstruction. In recent years, various specific priors which take advantage of Manhattan rule, geometry template and trained category features have been proposed to enhance the modeling performance. The advantages and limitations of both kinds of priors are discussed and evaluated in the paper. Finally, we discuss the trend and challenges of the prior studies in the future.

Blind optical aberration correction by exploring geometric and visual priors

Optical aberration widely exists in optical imaging systems, especially in consumer-level cameras. In contrast to previous solutions using hardware compensation or pre-calibration, we propose a computational approach for blind aberration removal from a single image, by exploring various geometric and visual priors. The global rotational symmetry allows us to transform the non-uniform degeneration into several uniform ones by the proposed radial splitting and warping technique. Locally, two types of symmetry constraints, i.e. central symmetry and reflection symmetry are defined as geometric priors in central and surrounding regions, respectively. Furthermore, by investigating the visual artifacts of aberration degenerated images captured by consumer-level cameras, the non-uniform distribution of sharpness across color channels and the image lattice is exploited as visual priors, resulting in a novel strategy to utilize the guidance from the sharpest channel and local image regions to improve the overall performance and robustness. Extensive evaluation on both real and synthetic data suggests that the proposed method outperforms the state-of-the-art techniques.

Image quality enhancement using original lens via optical computing

High-end lenses are usually composed of many optical elements to compensate various optical aberrations, e.g. geometric distortion, monochromatic and chromatic aberrations. The resulting complexity and machining accuracy requirements make high-end lenses too expensive, heavy, and fragile for day-to-day photography. To address this problem, we devised an optical computing approach to touch-up the low quality photos produced by simpler lenses. We propose a setup consisting of an easily accessible display and the original camera in order to perform optical aberration correction with a deconvolution framework. The equivalence of the degeneration model and the lenss optical computing turns the traditional blind deconvolution algorithm into its non-blind counterpart and promises robust performance. A prototype system is implemented to verify the feasibility of the proposed method, and a series of experiments on both synthetic and captured images are applied to demonstrate effectiveness and performance.

High-Dimensional Camera Shake Removal With Given Depth Map

Camera motion blur is drastically nonuniform for large depth-range scenes, and the nonuniformity caused by camera translation is depth dependent but not the case for camera rotations. To restore the blurry images of large-depth-range scenes deteriorated by arbitrary camera motion, we build an image blur model considering 6-degrees of freedom (DoF) of camera motion with a given scene depth map. To make this 6D depth-aware model tractable, we propose a novel parametrization strategy to reduce the number of variables and an effective method to estimate high-dimensional camera motion as well. The number of variables is reduced by temporal sampling motion function, which describes the 6-DoF camera motion by sampling the camera trajectory uniformly in time domain. To effectively estimate the high-dimensional camera motion parameters, we construct the probabilistic motion density function (PMDF) to describe the probability distribution of camera poses during exposure, and apply it as a unified constraint to guide the convergence of the iterative deblurring algorithm. Specifically, PMDF is computed through a back projection from 2D local blur kernels to 6D camera motion parameter space and robust voting. We conduct a series of experiments on both synthetic and real captured data, and validate that our method achieves better performance than existing uniform methods and nonuniform methods on large-depth-range scenes.

Optical Computing System for Fast Non-uniform Image Deblurring

Removing non-uniform blurring caused by camera shake has been troublesome for its high computational cost. To accelerate the non-uniform deblurring process, this paper analyzes the efficiency bottleneck of the non-uniform deblurring algorithms and proposes to implement the time-consuming and repeatedly required module, i.e. non-uniform convolution, by optical computing. Specifically, the non-uniform convolution is simulated by an off-the-shelf projector together with a camera mounted on a programmable motion platform. Benefiting from the high speed and parallelism of optical computation, our system is able to accelerate most existing non-uniform camera shake removing algorithms extensively. We develop a prototype system which can fast compute non-uniform convolution for the blurring image of planar scene caused by 3D rotation. By incorporating it into an iterative deblurring framework, the effectiveness of proposed system is verified.

Heterogeneous camera array for multispectral light field imaging

Multispectral light field acquisition is challenging due to the increased dimensionality of the problem. In this paper, inspired by anaglyph theory (i.e. the ability of human eyes to synthesize colored stereo perception from color-complementary (such as red and cyan) views), we propose to capture the multispectral light field using multiple cameras with different wide band filters. A convolutional neural network is used to extract the joint information of different spectral channels and to pair the cross-channel images. In our experiment, results on both synthetic data and real data captured by our prototype system validate the effectiveness and accuracy of proposed method.

Modeling the impact of spatial resolutions on perceptual quality of immersive image/video

We have attempted to investigate the impact of spatial resolutions on perceptual quality of immersive video/image contents rendered using a head mounted display. Subjective quality assessment is performed using the popular HTC Vive system. As demonstrated through the extensive experiments, spatial resolution impact on immersive image perceptual quality can be well described by an exponential model with a single model parameter, with averaged root mean squared errors (RMSE) less than 5% and Pearson correlation (PC) coefficient larger than 0.94. Model parameter characterizes the quality degradation speed when decreasing the spatial resolution, and now is derived using the least-squared-error fitting optimization.

High-resolution spectral video acquisition

Compared with conventional cameras, spectral imagers provide many more features in the spectral domain. They have been used in various fields such as material identification, remote sensing, precision agriculture, and surveillance. Traditional imaging spectrometers use generally scanning systems. They cannot meet the demands of dynamic scenarios. This limits the practical applications for spectral imaging. Recently, with the rapid development in computational photography theory and semiconductor techniques, spectral video acquisition has become feasible. This paper aims to offer a review of the state-of-the-art spectral imaging technologies, especially those capable of capturing spectral videos. Finally, we evaluate the performances of the existing spectral acquisition systems and discuss the trends for future work.