Suhas Sreehari

Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation

Many material and biological samples in scientific imaging are characterized by nonlocal repeating structures. These are studied using scanning electron microscopy and electron tomography. Sparse sampling of individual pixels in a two-dimensional image acquisition geometry, or sparse sampling of projection images with large tilt increments in a tomography experiment, can enable high speed data acquisition and minimize sample damage caused by the electron beam. In this paper, we present an algorithm for electron tomographic reconstruction and sparse image interpolation that exploits the nonlocal redundancy in images. We adapt a framework, termed plug-and-play priors, to solve these imaging problems in a regularized inversion setting. The power of the plug-and-play approach is that it allows a wide array of modern denoising algorithms to be used as a “prior model” for tomography and image interpolation. We also present sufficient mathematical conditions that ensure convergence of the plug-and-play approach, and we use these insights to design a new nonlocal means denoising algorithm. Finally, we demonstrate that the algorithm produces higher quality reconstructions on both simulated and real electron microscope data, along with improved convergence properties compared to other methods.

High frame-rate multichannel beam-scanning microscopy based on Lissajous trajectories

A simple beam-scanning optical design based on Lissajous trajectory imaging is described for achieving up to kHz frame-rate optical imaging on multiple simultaneous data acquisition channels. In brief, two fast-scan resonant mirrors direct the optical beam on a circuitous trajectory through the field of view, with the trajectory repeat-time given by the least common multiplier of the mirror periods. Dicing the raw time-domain data into sub-trajectories combined with model-based image reconstruction (MBIR) 3D in-painting algorithms allows for effective frame-rates much higher than the repeat time of the Lissajous trajectory. Since sub-trajectory and full-trajectory imaging are simply different methods of analyzing the same data, both high-frame rate images with relatively low resolution and low frame rate images with high resolution are simultaneously acquired. The optical hardware required to perform Lissajous imaging represents only a minor modification to established beam-scanning hardware, combined with additional control and data acquisition electronics. Preliminary studies based on laser transmittance imaging and polarization-dependent second harmonic generation microscopy support the viability of the approach both for detection of subtle changes in large signals and for trace-light detection of transient fluctuations.

Advanced prior modeling for 3D bright field electron tomography

Many important imaging problems in material science involve reconstruction of images containing repetitive non-local structures. Model-based iterative reconstruction (MBIR) could in principle exploit such redundancies through the selection of a log prior probability term. However, in practice, determining such a log prior term that accounts for the similarity between distant structures in the image is quite challenging. Much progress has been made in the development of denoising algorithms like non-local means and BM3D, and these are known to successfully capture non-local redundancies in images. But the fact that these denoising operations are not explicitly formulated as cost functions makes it unclear as to how to incorporate them in the MBIR framework. In this paper, we formulate a solution to bright field electron tomography by augmenting the existing bright field MBIR method to incorporate any non-local denoising operator as a prior model. We accomplish this using a framework we call plug-and-play priors that decouples the log likelihood and the log prior probability terms in the MBIR cost function. We specifically use 3D non-local means (NLM) as the prior model in the plug-and-play framework, and showcase high quality tomographic reconstructions of a simulated aluminum spheres dataset, and two real datasets of aluminum spheres and ferritin structures. We observe that streak and smear artifacts are visibly suppressed, and that edges are preserved. Also, we report lower RMSE values compared to the conventional MBIR reconstruction using qGGMRF as the prior model.

Plug-and-Play Unplugged: Optimization-Free Reconstruction Using Consensus Equilibrium

Regularized inversion methods for image reconstruction are used widely due to their tractability and ability to combine complex physical sensor models with useful regularity criteria. Such methods motivated the recently developed Plug-and-Play prior method, which provides a framework to use advanced denoising algorithms as regularizers in inversion. However, the need to formulate regularized inversion as the solution to an optimization problem limits the possible regularity conditions and physical sensor models. In this paper, we introduce Consensus Equilibrium (CE), which generalizes regularized inversion to include a much wider variety of both forward components and prior components without the need for either to be expressed with a cost function. CE is based on the solution of a set of equilibrium equations that balance data fit and regularity. In this framework, the problem of MAP estimation in regularized inversion is replaced by the problem of solving these equilibrium equations, which can be approached in multiple ways. The key contribution of CE is to provide a novel framework for fusing multiple heterogeneous models of physical sensors or models learned from data. We describe the derivation of the CE equations and prove that the solution of the CE equations generalizes the standard MAP estimate under appropriate circumstances. We also discuss algorithms for solving the CE equations, including ADMM with a novel form of preconditioning and Newtons method. We give examples to illustrate consensus equilibrium and the convergence properties of these algorithms and demonstrate this method on some toy problems and on a denoising example in which we use an array of convolutional neural network denoisers, none of which is tuned to match the noise level in a noisy image but which in consensus can achieve a better result than any of them individually.

Multi-Resolution Data Fusion for Super-Resolution Electron Microscopy

Perhaps surprisingly, all electron microscopy (EM) data collected to date is less than a cubic millimeter – presenting a huge demand in the materials and biological sciences to image at greater speed and lower dosage, while maintaining resolution. Traditional EM imaging based on homogeneous raster scanning severely limits the volume of high-resolution data that can be collected, and presents a fundamental limitation to understanding physical processes such as material deformation and crack propagation.,,,,,, We introduce a multi-resolution data fusion (MDF) method for super-resolution computational EM. Our method combines innovative data acquisition with novel algorithmic techniques to dramatically improve the resolution/ volume/speed trade-off. The key to our approach is to collect the entire sample at low resolution, while simultaneously collecting a small fraction of data at high resolution. The high-resolution measurements are then used to create a material-specific model that is used within the “plug-andplay” framework to dramatically improve resolution of the low-resolution data. We present results using FEI electron microscope data that demonstrate super-resolution factors of 4x-16x, while substantially maintaining high image quality and reducing dosage.

Sparse sampling image reconstruction in Lissajous trajectory beam-scanning multiphoton microscopy

Propagation of action potentials arises on millisecond timescales, suggesting the need for advancement of methods capable of commensurate volume rendering for in vivo brain mapping. In practice, beam-scanning multiphoton microscopy is widely used to probe brain function, striking a balance between simplicity and penetration depth. However, conventional beam-scanning platforms generally do not provide access to full volume renderings at the speeds necessary to map propagation of action potentials. By combining a sparse sampling strategy based on Lissajous trajectory microscopy in combination with temporal multiplexing for simultaneous imaging of multiple focal planes, whole volumes of cells are potentially accessible each millisecond.

Rotationally-invariant non-local means for image denoising and tomography

Many samples imaged in structural biology and material science contain several similar particles at random locations and orientations. Model-based iterative reconstruction (MBIR) methods can in principle be used to exploit such redundancies in images through log prior probabilities that accurately account for non-local similarity between the particles. However, determining such a log prior term can be challenging. Several denoising algorithms like non-local means (NLM) successfully capture such non-local redundancies, but the problem is two-fold: NLM is not explicitly formulated as a cost function, and neither can it capture similarity between randomly oriented particles. In this paper, we propose a rotationally-invariant nonlocal means (RINLM) algorithm, and describe a method to implement RINLM as a prior model using a novel framework that we call plug-and-play priors. We introduce the idea of patch pre-rotation to make RINLM computationally tractable. Finally, we showcase image denoising and 2D tomography results, using the proposed RINLM algorithm, as we highlight high reconstruction quality, image sharpness, and artifact suppression.

Multi-channel beam-scanning imaging at kHz frame rates by Lissajous trajectory microscopy.

A beam-scanning microscope based on Lissajous trajectory imaging is described for achieving streaming 2D imaging with continuous frame rates up to 1.4 kHz. The microscope utilizes two fast-scan resonant mirrors to direct the optical beam on a circuitous trajectory through the field of view. By separating the full Lissajous trajectory time-domain data into sub-trajectories (partial, undersampled trajectories) effective frame-rates much higher than the repeat time of the Lissajous trajectory are achieved with many unsampled pixels present. A model-based image reconstruction (MBIR) 3D in-painting algorithm is then used to interpolate the missing data for the unsampled pixels to recover full images. The MBIR algorithm uses a maximum a posteriori estimation with a generalized Gaussian Markov random field prior model for image interpolation. Because images are acquired using photomultiplier tubes or photodiodes, parallelization for multi-channel imaging is straightforward. Preliminary results show that when combined with the MBIR in-painting algorithm, this technique has the ability to generate kHz frame rate images across 6 total dimensions of space, time, and polarization for SHG, TPEF, and confocal reflective birefringence data on a multimodal imaging platform for biomedical imaging. The use of a multichannel data acquisition card allows for multimodal imaging with perfect image overlay. Image blur due to sample motion was also reduced by using higher frame rates.

Library-Based Sparse Interpolation and Super-Resolution of S/TEM Images of Biological and Material Nano-Structures

Scanning transmission electron microscopes are extensively used for characterization of biological and material samples at the nano-meter scale. However, raster scanning an electron beam across a large field of view is time consuming and can damage the sample. Additionally, in order to form large field of view raster scanned images in a reasonable amount of time, during which the instrument remains at optimal stability, spatial resolution is typically compromised and a standard raster scan size is used (e.g. 2048x2048 or 4094x4096). This leaves significant opportunities for super-resolution and sparse interpolation image reconstruction. For these reasons, there has been a growing need to accurately reconstruct images from sparsely sampled or low-resolution S/TEM images. In many cases, images of biological and material samples contain many structures that are similar or identical to each other. Inspired by the success of modern patch-based denoising filters like non-local means (NLM) in exploiting non-local image redundancies, there have been several efforts to solve the sparse interpolation and superresolution problems using patch-based models, dictionary learning and example-based methods [1, 2]. In any case, there is no general framework to use any generic denoising algorithm (with or without a patch library) as a prior model to perform sparse interpolation and super-resolution.

Model-Based Super-Resolution of SEM Images of Nano-Materials

Many imaging problems in materials and biological sciences involve reconstruction of highresolution images from low-resolution SEM images that contain several similar or identical nonlocal structures. Such SEM images can be sparsely represented owing to the enormous redundancy caused by repeating structures. Model-based iterative reconstruction (MBIR) is a powerful iterative reconstruction framework that could theoretically exploit such redundancies [1]. However, in practice, determining a prior probability term in the maximum a posteriori cost function that accounts for the similarity between non-local structures remains an open problem. Meanwhile, non-local patch-based denoising algorithms like non-local means (NLM) have been known to exploit non-local similarities in images. In fact, there have been several efforts to solve the super-resolution problem using patch-based models, dictionary learning and examplebased methods [2]. In any case, it is unclear how to use NLM-based denoising algorithms as prior models within the MBIR framework.