Han Lun Yap

Concentration of Measure for Block Diagonal Matrices With Applications to Compressive Signal Processing

Theoretical analysis of randomized, compressive operators often depends on a concentration of measure inequality for the operator in question. Typically, such inequalities quantify the likelihood that a random matrix will preserve the norm of a signal after multiplication. Concentration of measure results are well established for unstructured compressive matrices, populated with independent and identically distributed (i.i.d.) random entries. Many real-world acquisition systems, however, are subject to architectural constraints that make such matrices impractical. In this paper we derive concentration of measure bounds for two types of block diagonal compressive matrices, one in which the blocks along the main diagonal are random and independent, and one in which the blocks are random but equal. For both types of matrices, we show that the likelihood of norm preservation depends on certain properties of the signal being measured, but that for the best case signals, both types of block diagonal matrices can offer concentration performance on par with their unstructured, i.i.d. counterparts. We support our theoretical results with illustrative simulations as well as analytical and empirical investigations of several signal classes that are highly amenable to measurement using block diagonal matrices. We also discuss applications of these results in ensuring stable embeddings for various signal families and in establishing performance guarantees for solving various signal processing tasks (such as detection and classification) directly in the compressed domain.

The Restricted Isometry Property for block diagonal matrices

In compressive sensing (CS), the Restricted Isometry Property (RIP) is a powerful condition on measurement operators which ensures robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms. Early papers in CS showed that Gaussian random matrices satisfy the RIP with high probability, but such matrices are usually undesirable in practical applications due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures. To alleviate some or all of these difficulties, recent research efforts have focused on structured random matrices. In this paper, we study block diagonal measurement matrices where each block on the main diagonal is itself a Gaussian random matrix. The main result of this paper shows that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on the coherence of the basis in which the signals are sparse. In the best case—for signals that are sparse in the frequency domain—these matrices perform nearly as well as dense Gaussian random matrices despite having many fewer nonzero entries.

Short-term memory capacity in networks via the restricted isometry property

Cortical networks are hypothesized to rely on transient network activity to support short-term memory (STM). In this letter, we study the capacity of randomly connected recurrent linear networks for performing STM when the input signals are approximately sparse in some basis. We leverage results from compressed sensing to provide rigorous nonasymptotic recovery guarantees, quantifying the impact of the input sparsity level, the input sparsity basis, and the network characteristics on the system capacity. Our analysis demonstrates that network memory capacities can scale superlinearly with the number of nodes and in some situations can achieve STM capacities that are much larger than the network size. We provide perfect recovery guarantees for finite sequences and recovery bounds for infinite sequences. The latter analysis predicts that network STM systems may have an optimal recovery length that balances errors due to omission and recall mistakes. Furthermore, we show that the conditions yielding optimal STM capacity can be embodied in several network topologies, including networks with sparse or dense connectivities.

Concentration of measure for block diagonal measurement matrices

Concentration of measure inequalities are at the heart of much theoretical analysis of randomized compressive operators. Though commonly studied for dense matrices, in this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal blocks are i.i.d. subgaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the energy distribution of the signal plays in distinguishing the best case from the worst. We illustrate these phenomena with a series of experiments.

Stable Manifold Embeddings With Structured Random Matrices

The fields of compressed sensing (CS) and matrix completion have shown that high-dimensional signals with sparse or low-rank structure can be effectively projected into a low-dimensional space (for efficient acquisition or processing) when the projection operator achieves a stable embedding of the data by satisfying the Restricted Isometry Property (RIP). It has also been shown that such stable embeddings can be achieved for general Riemannian submanifolds when random orthoprojectors are used for dimensionality reduction. Due to computational costs and system constraints, the CS community has recently explored the RIP for structured random matrices (e.g., random convolutions, localized measurements, deterministic constructions). The main contribution of this paper is to show that any matrix satisfying the RIP (i.e., providing a stable embedding for sparse signals) can be used to construct a stable embedding for manifold-modeled signals by randomizing the column signs and paying reasonable additional factors in the number of measurements, thereby generalizing previous stable manifold embedding results beyond unstructured random matrices. We demonstrate this result with several new constructions for stable manifold embeddings using structured matrices. This result allows advances in efficient projection schemes for sparse signals to be immediately applied to manifold signal models.

Stable manifold embeddings with operators satisfying the Restricted Isometry Property

Signals of interests can often be thought to come from a low dimensional signal model. The exploitation of this fact has led to many recent interesting advances in signal processing, one notable example being in the field of compressive sensing (CS). The literature on CS has established that many matrices satisfy the Restricted Isometry Property (RIP), which guarantees a stable (i.e., distance-preserving) embedding of a sparse signal model from an undersampled linear measurement system. In this work, we study the stable embedding of manifold signal models using matrices that satisfy the RIP. We show that by paying reasonable additional factors in the number of measurements, all matrices that satisfy the RIP can also be used (in conjunction with a random sign sequence) to obtain a stable embedding of a manifold.

Concentration of measure for block diagonal matrices with repeated blocks

The theoretical analysis of randomized compressive operators often relies on the existence of a concentration of measure inequality for the operator of interest. Though commonly studied for unstructured, dense matrices, matrices with more structure are often of interest because they model constraints on the sensing system or allow more efficient system implementations. In this paper we derive a concentration of measure bound for block diagonal matrices where the nonzero entries along the main diagonal are a single repeated block of i.i.d. Gaussian random variables. Our main result states that the concentration exponent, in the best case, scales as that for a fully dense matrix. We also identify the role that the signal diversity plays in distinguishing the best and worst cases. Finally, we illustrate these phenomena with a series of experiments.

The Restricted Isometry Property for Echo State Networks with applications to sequence memory capacity

The ability of networked systems (including artificial or biological neuronal networks) to perform complex data processing tasks relies in part on their ability to encode signals from the recent past in the current network state. Here we use Compressed Sensing tools to study the ability of a particular network architecture (Echo State Networks) to stably store long input sequences. In particular, we show that such networks satisfy the Restricted Isometry Property when the input sequences are compressible in certain bases and when the number of nodes scale linearly with the sparsity of the input sequence and logarithmically with its dimension. Thus, the memory capacity of these networks depends on the input sequence statistics, and can (sometimes greatly) exceed the number of nodes in the network. Furthermore, input sequences can be robustly recovered from the instantaneous network state using a tractable optimization program (also implementable in a network architecture).

Poutine: A correlation estimator for ergodic stationary signals

In this work, we present POUTINE, a novel estimator of the auto-correlation function (or more generally, the cross-correlation function) of ergodic stationary signals, an important task in a variety of applications. This estimator sparsely and non-adaptively samples the process via Bernoulli selection, generalizing the classical estimator in a natural way, and offering significant sampling reductions while sacrificing a modest degree of accuracy. Both the mean and variance of our estimator are explicitly analyzed, and in particular, we show that POUTINE gives an unbiased estimate of the classical estimator, which in turn gives an unbiased estimate of the underlying second-order statistics of interest. Furthermore, we show that POUTINE is a consistent estimator with variance approaching zero asymptotically. We demonstrate favorable performance of this approach for a simple stochastic process.