G. M. Dilshan Godaliyadda

A Supervised Learning Approach for Dynamic Sampling.

Sparse sampling schemes have the potential to reduce image acquisition time by reconstructing a desired image from a sparse subset of measured pixels. Moreover, dynamic sparse sampling methods have the greatest potential because each new pixel is selected based on information obtained from previous samples. However, existing dynamic sampling methods tend to be computationally expensive and therefore too slow for practical application. In this paper, we present a supervised learning based algorithm for dynamic sampling (SLADS) that uses machine-learning techniques to select the location of each new pixel measurement. SLADS is fast enough to be used in practical imaging applications because each new pixel location is selected using a simple regression algorithm. In addition, SLADS is accurate because the machine learning algorithm is trained using a total reduction in distortion metric which accounts for distortion in a neighborhood of the pixel being sampled. We present results on both computationally-generated synthetic data and experimentallycollected data that demonstrate substantial improvement relative to state-of-the-art static sampling methods. Introduction In conventional point-wise image acquisition, all pixels in a rectilinear grid are measured. However, in many imaging applications, a high-fidelity pixel measurement could take up to 1 second. Examples of such methods include electron back scatter diffraction (EBSD) microscopy and Raman spectroscopy, which are of great importance in material science and chemistry [1]. Then, acquiring a complete set of high-resolution measurements on these imaging applications becomes impractical. Sparse sampling offers the potential to dramatically reduce the time required to acquire an image. In this approach, a sparse set of pixels is measured, and the full resolution image is reconstructed from the set of sparse measurements. In addition to speeding image acquisition, sparse sampling methods also hold the potential to reduce the exposure of the object being imaged to destructive radiation. This is of critical importance when imaging biological samples using X-rays, electrons, or even optical photons [2, 3]. Sparse sampling approaches fall into two main categories: static and dynamic. In static sampling, pixels are measured in a pre-defined order. Examples of static sparse sampling methods include random sampling strategies such as in [4], and lowdiscrepancy sampling [5]. As a result some samples from these methods may not be very informative, as they do not take into account the object being scanned. There are static sampling methods based on an a priori knowledge of the object geometry and sparsity such as [6, 7]. However a priori knowledge is not always available for general imaging applications On the other hand, dynamic sampling (DS) methods adaptively determine new measurement locations based on the information obtained from previous measurements. This is a very powerful technique since in real applications previous measurements can tell one a great deal about the object being scanned and also about the best locations for future measurements Therefore, dynamic sampling has the potential to dramatically reduce the total number of samples required to achieve a particular level of distortion in the reconstructed image. An example of a dynamic sampling method was proposed in [8] by Kovačević et al. Here initially an object is measured with a sparse grid. Then, if the intensity of a pixel is above a certain threshold, the vicinity of that pixel is measured in higher resolution. However, the threshold was empirically chosen for the specific scanner and thus this method cannot be generalized for different imaging modalities. For general applications, a set of DS methods has been proposed in previous literature where an objective function is designed and the measurements are chosen to optimize that objective function. For instance, dynamic compressive sensing methods [9–11] find the next measurements that maximally reduces the differential entropy. However, dynamic compressive sensing methods use an unconstrained projection as a measurement and therefore are not suitable for point-wise measurements where the measurement is constrained. Apart from these methods, application specific DS methods that optimize an objective function to find the next measurement have been developed. One example is [12], where the authors modify the optimal experimental design [13] framework to incorporate dynamic measurement selection in a biochemical network. Seeger et al. in [14] also finds the measurement that reduces the differential entropy the most but now to select optimal K-space spiral and line measurements for magnetic resonance imaging (MRI). In addition, Batenburg et al. [15] propose a DS method for binary computed tomography in which the measurement that maximizes the information gain is selected. Even though these measurements are constrained they are application specific and therefore not applicable to general point-wise measurements. In [16] Godaliyadda et al. propose a DS algorithm for general point-wise measurements. Here, the authors use a MonteCarlo simulation method to approximate the conditional variance at every unmeasured location, given previous measurements, and select the pixel with largest conditional variance. However, Monte-Carlo simulation methods such as the Metropolis-Hastings method are very slow and therefore this method is infeasible for real-time applications. Furthermore, the objective function in this method does not account for the change of conditional variance in ©2016 Society for Imaging Science and Technology DOI: 10.2352/ISSN.2470-1173.2016.19.COIMG-153 IS&T International Symposium on Electronic Imaging 2016 Computational Imaging XIV COIMG-153.1 the entire image with a new measurement. In this paper, we propose a new DS algorithm for point-wise measurements named supervised learning approach for dynamic sampling (SLADS). The objective of SLADS is to select a new pixel so as to maximally reduce the conditional expectation of the reduction in distortion (ERD) in the entire reconstructed image. In SLADS, we compute the reduction in distortion for each pixel in a training data set, and then find the relationship between the ERD and a local feature vector through a regression algorithm. Since we use a supervised learning approach, we can very rapidly estimate the ERD at each pixel in the unknown testing image. Moreover, we introduce a measure that approximates the distortion reduction in the training dataset so that it accounts for the distortion reduction in the pixel and its neighbors. Since computing the distortion reduction for each pixel during training can be intractable, particularly for large images, this approximation is vital to make the training procedure feasible. Experimental results on sampling a computationally-generated synthetic EBSD image and an experimentally-collected image have shown that SLADS can compute a new sample locations very quickly (in the range of 5 500 ms), and can achieve the same reconstruction distortion as static sampling methods with dramatically fewer samples (2-4 times fewer). Dynamic Sampling Framework The objective in sparse sampling is to measure a sparse set of pixels in an image and then reconstruct the full resolution image from those sparse samples. Moreover, with sparse dynamic sampling, the location for each new pixel to measure will be informed by all the previous pixel measurements. To formulate the problem, we denote the image we would like to measure as X ∈ RN , where Xr is a pixel at location r ∈ Ω. Furthermore, let us assume that k pixels have been measured at a set of locations S = {s(1), · · · ,s(k)}, and that the corresponding measured values and locations are represented by the k×2 matrix Y (k) =  s,Xs(1) .. s,Xs(k)  . Then from Y (k), we can reconstruct an image X̂ (k), which is our best estimate of X given the first k measurements. Now, if we select Xs as our next pixel to measure, then presumably we can reconstruct a better estimate of the image, which we will denote by X̂ (k;s). So then X̂ (k;s) is our best estimate of X given both Y (k) and Xs. So at this point, our goal is to select the next location s(k+1) that results in the greatest decrease in reconstruction distortion. In order to formulate this problem, let D(Xr, X̂r) denote the distortion measure between a pixel Xr and its estimate X̂r, and let D(X , X̂) = ∑ r∈Ω D(Xr, X̂r) , (1) denote the total distortion between the image X and its estimate X̂ . Then using this notation, we may define R r to be the local reduction in distortion at pixel r that would result from the measurement of the pixel Xs. R r = D(Xr, X̂ (k) r )−D(Xr, X̂ (k;s) r ) (2) Importantly, the measurement of the pixel Xs does not only reduce distortion at that pixel. It also reduces the distortion at neighboring pixels. So in order to represent the total reduction in distortion, we must sum over all pixels r ∈Ω. R(k;s) = ∑ r∈Ω R r (3) = D(X , X̂ (k))−D(X , X̂ (k;s)) . (4) Now of course, we do not know what the value of Xs until it is measured; so we also do not know the value R(k;s). Therefore, we must make our selection of the next pixel based on the conditional expectation of reduction in distortion which we will refer to as the ERD given by R̄(k;s) = E [ R(k;s)|Y (k) ] . (5) So with this notation, our goal is to efficiently compute the next pixel to sample, s(k+1), as the solution to the following optimization. s(k+1) = arg max s∈{Ω\S } ( R̄(k;s) ) (6) Once we measure the location Xs(k+1) , then we form the new measurement vector Y (k+1) = [ Y (k) s,Xs(k+1) ]

Dynamic X-ray diffraction sampling for protein crystal positioning

A sparse supervised learning approach for dynamic sampling (SLADS) is described for dose reduction in diffraction-based protein crystal positioning. Crystal centering is typically a prerequisite for macromolecular diffraction at synchrotron facilities, with X-ray diffraction mapping growing in popularity as a mechanism for localization. In X-ray raster scanning, diffraction is used to identify the crystal positions based on the detection of Bragg-like peaks in the scattering patterns; however, this additional X-ray exposure may result in detectable damage to the crystal prior to data collection. Dynamic sampling, in which preceding measurements inform the next most information-rich location to probe for image reconstruction, significantly reduced the X-ray dose experienced by protein crystals during positioning by diffraction raster scanning. The SLADS algorithm implemented herein is designed for single-pixel measurements and can select a new location to measure. In each step of SLADS, the algorithm selects the pixel, which, when measured, maximizes the expected reduction in distortion given previous measurements. Ground-truth diffraction data were obtained for a 5 µm-diameter beam and SLADS reconstructed the image sampling 31% of the total volume and only 9% of the interior of the crystal greatly reducing the X-ray dosage on the crystal. Using in situ two-photon-excited fluorescence microscopy measurements as a surrogate for diffraction imaging with a 1 µm-diameter beam, the SLADS algorithm enabled image reconstruction from a 7% sampling of the total volume and 12% sampling of the interior of the crystal. When implemented into the beamline at Argonne National Laboratory, without ground-truth images, an acceptable reconstruction was obtained with 3% of the image sampled and approximately 5% of the crystal. The incorporation of SLADS into X-ray diffraction acquisitions has the potential to significantly minimize the impact of X-ray exposure on the crystal by limiting the dose and area exposed for image reconstruction and crystal positioning using data collection hardware present in most macromolecular crystallography end-stations.

A model-based framework for fast dynamic image sampling

In many applications, it is critical to be able to sample the most informative pixels of an image first; and then once these pixels are sampled, the highest fidelity image can be reconstructed. Optimized sampling strategies generally fall into two categories: static and dynamic. In dynamic sampling, each new sample is chosen by using information obtained from previous samples. In this way, dynamic sampling offers the potential of much greater fidelity, but at the cost of greater complexity. Existing methods for dynamic non-uniform sampling of images are based on the intuition that sampling rates should be greatest in locations of greatest variation, but recent developments in the theory of optimal experimental design offer a theoretical framework for optimal sampling based on the use of a formal Bayesian prior model. In this paper, we introduce a fast dynamic image sampling framework based on Bayesian experimental design (BED). The method, which we call model-based dynamic sampling (MBDS) allows for the use of a general prior distribution for the image, and it incorporates a pixel-wise sampling constraint in the BED framework. The MBDS works by first generating L stochastic samples (i.e., images) from the posterior distribution given the current measurements, and then selecting the pixel with the greatest posterior variance. We also introduce a computationally efficient method for computing the stochastic samples through a local updating technique.

Reduced electron exposure for energy-dispersive spectroscopy using dynamic sampling

Analytical electron microscopy and spectroscopy of biological specimens, polymers, and other beam sensitive materials has been a challenging area due to irradiation damage. There is a pressing need to develop novel imaging and spectroscopic imaging methods that will minimize such sample damage as well as reduce the data acquisition time. The latter is useful for high-throughput analysis of materials structure and chemistry. In this work, we present a novel machine learning based method for dynamic sparse sampling of EDS data using a scanning electron microscope. Our method, based on the supervised learning approach for dynamic sampling algorithm and neural networks based classification of EDS data, allows a dramatic reduction in the total sampling of up to 90%, while maintaining the fidelity of the reconstructed elemental maps and spectroscopic data. We believe this approach will enable imaging and elemental mapping of materials that would otherwise be inaccessible to these analysis techniques.

Dynamic Sparse Sampling for Confocal Raman Microscopy

The total number of data points required for image generation in Raman microscopy was greatly reduced using sparse sampling strategies, in which the preceding set of measurements informed the next most information-rich sampling location. Using this approach, chemical images of pharmaceutical materials were obtained with >99% accuracy from 15.8% sampling, representing an ∼6-fold reduction in measurement time relative to full field of view rastering with comparable image quality. This supervised learning approach to dynamic sampling (SLADS) has the distinct advantage of being directly compatible with standard confocal Raman instrumentation. Furthermore, SLADS is not limited to Raman imaging, potentially providing time-savings in image reconstruction whenever the single-pixel measurement time is the limiting factor in image generation.

Under-sampling and Image Reconstruction for Scanning Electron Microscopes

Electron Microscopes have been used to investigate materials from micron to nano scale. Scanning electron microscopes (SEM) as well as scanning transmission electron microscopes (STEM) can acquire image data relatively fast, however acquiring spectroscopic data requires longer data collection times. Depending on the desired resolution or sample area, this can make a significant difference in the duration and feasibility of the experiment. Moreover, for electron beam sensitive samples, it is necessary to acquire the image data with minimal exposure time as not to further damage the sample [1]. Here, we propose an under-sampling and reconstruction method to reduce the data collection time while maintaining imaging accuracy.