Tomer Michaeli

Blind Deblurring Using Internal Patch Recurrence

Recurrence of small image patches across different scales of a natural image has been previously used for solving ill-posed problems (e.g. super- resolution from a single image). In this paper we show how this multi-scale property can also be used for “blind-deblurring”, namely, removal of an unknown blur from a blurry image. While patches repeat ‘as is’ across scales in a sharp natural image, this cross-scale recurrence significantly diminishes in blurry images. We exploit these deviations from ideal patch recurrence as a cue for recovering the underlying (unknown) blur kernel. More specifically, we look for the blur kernel k, such that if its effect is “undone” (if the blurry image is deconvolved with k), the patch similarity across scales of the image will be maximized. We report extensive experimental evaluations, which indicate that our approach compares favorably to state-of-the-art blind deblurring methods, and in particular, is more robust than them.

Beyond bandlimited sampling

Digital applications have developed rapidly over the last few decades. Since many sources of information are of analog or continuous-time nature, discrete-time signal processing (DSP) inherently relies on sampling a continuous-time signal to obtain a discrete-time representation. Consequently, sampling theories lie at the heart of signal processing devices and communication systems. Examples include sampling rate conversion for software radio and between audio formats, biomedical imaging, lens distortion correction and the formation of image mosaics, and super-resolution of image sequences.

Performance Bounds and Design Criteria for Estimating Finite Rate of Innovation Signals

In this paper, we consider the problem of estimating finite rate of innovation (FRI) signals from noisy measurements, and specifically analyze the interaction between FRI techniques and the underlying sampling methods. We first obtain a fundamental limit on the estimation accuracy attainable regardless of the sampling method. Next, we provide a bound on the performance achievable using any specific sampling approach. Essential differences between the noisy and noise-free cases arise from this analysis. In particular, we identify settings in which noise-free recovery techniques deteriorate substantially under slight noise levels, thus quantifying the numerical instability inherent in such methods. This instability, which is only present in some families of FRI signals, is shown to be related to a specific type of structure, which can be characterized by viewing the signal model as a union of subspaces. Finally, we develop a methodology for choosing the optimal sampling kernels for linear reconstruction, based on a generalization of the Karhunen-Loève transform. The results are illustrated for several types of time-delay estimation problems.

Xampling at the Rate of Innovation

We address the problem of recovering signals from samples taken at their rate of innovation. Our only assumption is that the sampling system is such that the parameters defining the signal can be stably determined from the samples, a condition that lies at the heart of every sampling theorem. Consequently, our analysis subsumes previously studied nonlinear acquisition devices and nonlinear signal classes. In particular, we do not restrict attention to memoryless nonlinear distortions or to union-of-subspace models. This allows treatment of various finite-rate-of-innovation (FRI) signals that were not previously studied, including, for example, continuous phase modulation transmissions. Our strategy relies on minimizing the error between the measured samples and those corresponding to our signal estimate. This least-squares (LS) objective is generally nonconvex and might possess many local minima. Nevertheless, we prove that under the stability hypothesis, any optimization method designed to trap a stationary point of the LS criterion necessarily converges to the true solution. We demonstrate our approach in the context of recovering pulse streams in settings that were not previously treated. Furthermore, in situations for which other algorithms are applicable, we show that our method is often preferable in terms of noise robustness.

U-Invariant Sampling: Extrapolation and Causal Interpolation From Generalized Samples

Causal processing of a signals samples is crucial in on-line applications such as audio rate conversion, compression, tracking and more. This paper addresses the problems of predicting future samples and causally interpolating deterministic signals. We treat a rich variety of sampling mechanisms encountered in practice, namely in which each sampling function is obtained by applying a unitary operator on its predecessor. Examples include pointwise sampling at the output of an antialiasing filter and magnetic resonance imaging (MRI), which correspond respectively to the translation and modulation operators. From an abstract Hilbert-space viewpoint, such sequences of functions were studied extensively in the context of stationary random processes. We thus utilize powerful tools from this discipline, although our problems are deterministic by nature. In particular, we provide necessary and sufficient conditions on the sampling mechanism such that perfect prediction is possible. For cases where perfect prediction is impossible, we derive the predictor minimizing the prediction error. We also derive a causal interpolation method that best approximates the commonly used noncausal solution. Finally, we study when causal processing of the samples of a signal can be performed in a stable manner.

High-Rate Interpolation of Random Signals From Nonideal Samples

We address the problem of reconstructing a random signal from samples of its filtered version using a given interpolation kernel. In order to reduce the mean squared error (MSE) when using a nonoptimal kernel, we propose a high rate interpolation scheme in which the interpolation grid is finer than the sampling grid. A digital correction system that processes the samples prior to their multiplication with the shifts of the interpolation kernel is developed. This system is constructed such that the reconstructed signal is the linear minimum MSE (LMMSE) estimate of the original signal given its samples. An analytic expression for the MSE as a function of the interpolation rate is provided, which leads to an explicit condition such that the optimal MSE is achieved with the given nonoptimal kernel. Simulations confirm the reduction in MSE with respect to a system with equal sampling and reconstruction rates.

Hidden Relationships: Bayesian Estimation With Partial Knowledge

We address the problem of Bayesian estimation where the statistical relation between the signal and measurements is only partially known. We propose modeling partial Bayesian knowledge by using an auxiliary random vector called instrument. The statistical relations between the instrument and the signal and between the instrument and the measurements, are known. However, the joint probability function of the signal and measurements is unknown. Two types of statistical relations are considered, corresponding to second-order moment and complete distribution function knowledge. We propose two approaches for estimation in partial knowledge scenarios. The first is based on replacing the orthogonality principle by an oblique counterpart and is proven to coincide with the method of instrumental variables from statistics, although developed in a different context. The second is based on a worst-case design strategy and is shown to be advantageous in many aspects. We provide a thorough analysis showing in which situations each of the methods is preferable and propose a nonparametric method for approximating the estimators from a set of examples. Finally, we demonstrate our approach in the context of enhancement of facial images that have undergone unknown degradation and image zooming.

LMMSE Filtering in Feedback Systems With White Random Modes: Application to Tracking in Clutter

A generalized state space representation of dynamical systems with random modes switching according to a white random process is presented. The new formulation includes a term, in the dynamics equation, that depends on the most recent linear minimum mean squared error (LMMSE) estimate of the state. This can model the behavior of a feedback control system featuring a state estimator. The measurement equation is allowed to depend on the previous LMMSE estimate of the state, which can represent the fact that measurements are obtained from a validation window centered about the predicted measurement and not from the entire surveillance region. The LMMSE filter is derived for the considered problem. The approach is demonstrated in the context of target tracking in clutter and is shown to be competitive with several popular nonlinear methods.

Partially Linear Estimation With Application to Sparse Signal Recovery From Measurement Pairs

We address the problem of estimating a random vector from two sets of measurements and , such that the estimator is linear in . We show that the partially linear minimum mean-square error (PLMMSE) estimator does not require knowing the joint distribution of and in full, but rather only its second-order moments. This renders it of potential interest in various applications. We further show that the PLMMSE method is minimax-optimal among all estimators that solely depend on the second-order statistics of and . We demonstrate our approach in the context of recovering a signal, which is sparse in a unitary dictionary, from noisy observations of it and of a filtered version. We show that in this setting PLMMSE estimation has a clear computational advantage, while its performance is comparable to state-of-the-art algorithms. We apply our approach both in static and in dynamic estimation applications. In the former category, we treat the problem of image enhancement from blurred/noisy image pairs. We show that PLMMSE estimation performs only slightly worse than state-of-the art algorithms, while running an order of magnitude faster. In the dynamic setting, we provide a recursive implementation of the estimator and demonstrate its utility in tracking maneuvering targets from position and acceleration measurements.

Deep-STORM: super-resolution single-molecule microscopy by deep learning

We present an ultra-fast, precise, parameter-free method, which we term Deep-STORM, for obtaining super-resolution images from stochastically-blinking emitters, such as fluorescent molecules used for localization microscopy. Deep-STORM uses a deep convolutional neural network that can be trained on simulated data or experimental measurements, both of which are demonstrated. The method achieves state-of-the-art resolution under challenging signal-to-noise conditions and high emitter densities, and is significantly faster than existing approaches. Additionally, no prior information on the shape of the underlying structure is required, making the method applicable to any blinking data-set. We validate our approach by super-resolution image reconstruction of simulated and experimentally obtained data.