Karin Schnass

Compressed Sensing and Redundant Dictionaries

This paper extends the concept of compressed sensing to signals that are not sparse in an orthonormal basis but rather in a redundant dictionary. It is shown that a matrix, which is a composition of a random matrix of certain type and a deterministic dictionary, has small restricted isometry constants. Thus, signals that are sparse with respect to the dictionary can be recovered via basis pursuit (BP) from a small number of random measurements. Further, thresholding is investigated as recovery algorithm for compressed sensing, and conditions are provided that guarantee reconstruction with high probability. The different schemes are compared by numerical experiments.

Dictionary Preconditioning for Greedy Algorithms

This paper introduces the concept of sensing dictionaries. It presents an alteration of greedy algorithms like thresholding or (orthogonal) matching pursuit which improves their performance in finding sparse signal representations in redundant dictionaries while maintaining the same complexity. These algorithms can be split into a sensing and a reconstruction step, and the former will fail to identify correct atoms if the cumulative coherence of the dictionary is too high. We thus modify the sensing step by introducing a special sensing dictionary. The correct selection of components is then determined by the cross cumulative coherence which can be considerably lower than the cumulative coherence. We characterize the optimal sensing matrix and develop a constructive method to approximate it. Finally, we compare the performance of thresholding and OMP using the original and modified algorithms.

Dictionary Identification—Sparse Matrix-Factorization via

This paper treats the problem of learning a dictionary providing sparse representations for a given signal class, via ℓ₁-minimization. The problem can also be seen as factorizing a d × N matrix Y = (y₁ . . . y_N), y_n ∈ ℝ^d of training signals into a d × K dictionary matrix Φ and a K × N coefficient matrix X = (x₁ . . . x_N), x_n ∈ ℝ^K, which is sparse. The exact question studied here is when a dictionary coefficient pair (Φ, X) can be recovered as local minimum of a (nonconvex) ℓ₁-criterion with input Y = Φ X. First, for general dictionaries and coefficient matrices, algebraic conditions ensuring local identifiability are derived, which are then specialized to the case when the dictionary is a basis. Finally, assuming a random Bernoulli-Gaussian sparse model on the coefficient matrix, it is shown that sufficiently incoherent bases are locally identifiable with high probability. The perhaps surprising result is that the typically sufficient number of training samples N grows up to a logarithmic factor only linearly with the signal dimension, i.e., N ≈ CK log K, in contrast to previous approaches requiring combinatorially many samples.

\ell_1

In this letter, we show that with high probability, the thresholding algorithm can recover signals that are sparse in a redundant dictionary as long as the 2-Babel function is growing slowly. This implies that it can succeed for sparsity levels up to the order of the ambient dimension. The theoretical bounds are illustrated with numerical simulations. As an application of the theory, sensing dictionaries for optimal average performance are characterized, and their performance is tested numerically.

-Minimization

Abstract This article gives theoretical insights into the performance of K-SVD, a dictionary learning algorithm that has gained significant popularity in practical applications. The particular question studied here is when a dictionary Φ ∈ R d × K can be recovered as local minimum of the minimisation criterion underlying K-SVD from a set of N training signals y n = Φ x n . A theoretical analysis of the problem leads to two types of identifiability results assuming the training signals are generated from a tight frame with coefficients drawn from a random symmetric distribution. First, asymptotic results showing that in expectation the generating dictionary can be recovered exactly as a local minimum of the K-SVD criterion if the coefficient distribution exhibits sufficient decay. Second, based on the asymptotic results it is demonstrated that given a finite number of training samples N, such that N / log ⁡ N = O ( K 3 d ) , except with probability O ( N − K d ) there is a local minimum of the K-SVD criterion within distance O ( K N − 1 / 4 ) to the generating dictionary.

Average Performance Analysis for Thresholding

This paper introduces p-thresholding, an algorithm to compute simultaneous sparse approximations of multichannel signals over redundant dictionaries. We work out both worst case and average case recovery analyses of this algorithm and show that the latter results in much weaker conditions on the dictionary. Numerical simulations confirm our theoretical findings and show that p-thresholding is an interesting low complexity alternative to simultaneous greedy or convex relaxation algorithms for processing sparse multichannel signals with balanced coefficients.

On the identifiability of overcomplete dictionaries via the minimisation principle underlying K-SVD

Many recent works have shown that if a given signal admits a sufficiently sparse representation in a given dictionary, then this representation is recovered by several standard optimization algorithms, in particular the convex l1 minimization approach. Here we investigate the related problem of inferring the dictionary from training data, with an approach where l1- minimization is used as a criterion to select a dictionary. We restrict our analysis to basis learning and identify necessary / sufficient / necessary and sufficient conditions on ideal (not necessarily very sparse) coefficients of the training data in an ideal basis to guarantee that the ideal basis is a strict local optimum of the A -minimization criterion among (not necessarily orthogonal) bases of normalized vectors. We illustrate these conditions on deterministic as well as toy random models in dimension two and highlight the main challenges that remain open by this preliminary theoretical results.

Average Case Analysis of Multichannel Thresholding

This paper presents a simple randomised algorithm for recovering high-dimensional sparse functions, i.e. functions ƒ : [0, 1]^d → ℝ which depend effectively only on k out of d variables, meaning ƒ(x<inf>1</inf>, …, x<inf>d</inf>) = g(x<inf>i1</inf>, …, x<inf>ik</inf> ), where the indices 1 ≤ i<inf>1</inf> &#60; i<inf>2</inf> &#60; … &#60; i<inf>k</inf> ≤ d are unknown. It is shown that (under certain conditions on g) this algorithm recovers the k unknown coordinates with probability at least 1–6 exp(−L) using only O(k(L+log k)(L+log d)) samples of ƒ.

Some recovery conditions for basis learning by L1-minimization

We present a new and computationally efficient scheme for classifying signals into a fixed number of known classes. We model classes as subspaces in which the corresponding data is well represented by a dictionary of features. In order to ensure low misclassification, the subspaces should be incoherent so that features of a given class cannot represent efficiently signals from another. We propose a simple iterative strategy to learn dictionaries which are are the same time good for approximating within a class and also discriminant. Preliminary tests on a standard face images database show competitive results.

Compressed learning of high-dimensional sparse functions

In this article we present a signal model for classification based on a low dimensional dictionary embedded into the high dimensional signal space. We develop an alternate projection algorithm to find the embedding and the dictionary and finally test the classification performance of our scheme in comparison to Fishers LDA.