Aurele Balavoine

Convergence and Rate Analysis of Neural Networks for Sparse Approximation

We present an analysis of the Locally Competitive Algorithm (LCA), which is a Hopfield-style neural network that efficiently solves sparse approximation problems (e.g., approximating a vector from a dictionary using just a few nonzero coefficients). This class of problems plays a significant role in both theories of neural coding and applications in signal processing. However, the LCA lacks analysis of its convergence properties, and previous results on neural networks for nonsmooth optimization do not apply to the specifics of the LCA architecture. We show that the LCA has desirable convergence properties, such as stability and global convergence to the optimum of the objective function when it is unique. Under some mild conditions, the support of the solution is also proven to be reached in finite time. Furthermore, some restrictions on the problem specifics allow us to characterize the convergence rate of the system by showing that the LCA converges exponentially fast with an analytically bounded convergence rate. We support our analysis with several illustrative simulations.

Hardware and software infrastructure for a family of floating-gate based FPAAs

Analog circuits and systems research and education can benefit from the flexibility provided by large-scale Field Programmable Analog Arrays (FPAAs). This paper presents the hardware and software infrastructure supporting the use of a family of floating-gate based FPAAs being developed at Georgia Tech. This infrastructure is compact and portable and provides the user with a comprehensive set of tools for custom analog circuit design and implementation. The infrastructure includes the FPAA IC, discrete ADC, DAC and amplifier ICs, a 32-Bit ARM based microcontroller for interfacing the FPAA with the users computer, and Matlab and targeting software. The FPAA hardware communicates with Matlab over a USB connection. The USB connection also provides the hardwares power. The software tools include three major systems: a Matlab Simulink FPAA program, a SPICE to FPAA compiler called GRASPER, and a visualization tool called RAT. The hardware consists of two custom PCB designs which include a main board used to program and control an FPAA IC and an FPAA IC adaptor board used to interface a QFP packaged FPAA IC with the 100 pin ZIF socket on the main programming and control board.

Convergence Speed of a Dynamical System for Sparse Recovery

This paper studies the convergence rate of a continuous-time dynamical system for l1-minimization, known as the Locally Competitive Algorithm (LCA). Solving l1-minimization problems efficiently and rapidly is of great interest to the signal processing community, as these programs have been shown to recover sparse solutions to underdetermined systems of linear equations and come with strong performance guarantees. The LCA under study differs from the typical l1-solver in that it operates in continuous time: instead of being specified by discrete iterations, it evolves according to a system of nonlinear ordinary differential equations. The LCA is constructed from simple components, giving it the potential to be implemented as a large-scale analog circuit. The goal of this paper is to give guarantees on the convergence time of the LCA system. To do so, we analyze how the LCA evolves as it is recovering a sparse signal from underdetermined measurements. We show that under appropriate conditions on the measurement matrix and the problem parameters, the path the LCA follows can be described as a sequence of linear differential equations, each with a small number of active variables. This allows us to relate the convergence time of the system to the restricted isometry constant of the matrix. Interesting parallels to sparse-recovery digital solvers emerge from this study. Our analysis covers both the noisy and noiseless settings and is supported by simulation results.

Global convergence of the Locally Competitive Algorithm

The Locally Competitive Algorithm (LCA) is a continuoustime dynamical system designed to solve the problem of sparse approximation. This class of approximation problems plays an important role in producing state-of-the-art results in many signal processing and inverse problems, and implementing the LCA in analog VLSI may significantly improve the time and power necessary to solve these optimization programs. The goal of this paper is to analyze the dynamical behavior of the LCA system and guarantee its convergence and stability. We show that fixed points of the system are extrema of the sparse approximation objective function when designed for a certain class of sparsity-inducing cost penalty. We also show that, if the objective has a unique minimum, the LCA converges for any initial point. In addition, we prove that under certain conditions on the solution, the LCA converges in a finite number of switches (i.e., node threshold crossings).

Discrete and Continuous-Time Soft-Thresholding for Dynamic Signal Recovery

There exist many well-established techniques to recover sparse signals from compressed measurements with known performance guarantees in the static case. More recently, new methods have been proposed to tackle the recovery of time-varying signals, but few benefit from a theoretical analysis. In this paper, we give theoretical guarantees for the Iterative Soft-Thresholding Algorithm (ISTA) and its continuous-time analogue the Locally Competitive Algorithm (LCA) to perform this tracking in real time. ISTA is a well-known digital solver for static sparse recovery, whose iteration is a first-order discretization of the LCA differential equation. Our analysis is based on the Restricted Isometry Property (RIP) and shows that the outputs of both algorithms can track a time-varying signal while compressed measurements are streaming, even when no convergence criterion is imposed at each time step. The l2-distance between the target signal and the outputs of both discrete- and continuous-time solvers is shown to decay to a bound that is essentially optimal. Our analysis is supported by simulations on both synthetic and real data.

Dynamic Filtering of Time-Varying Sparse Signals via

Despite the importance of sparsity signal models and the increasing prevalence of high-dimensional streaming data, there are relatively few algorithms for dynamic filtering of timevarying sparse signals. Of the existing algorithms, fewer still provide strong performance guarantees. This paper examines two algorithms for dynamic filtering of sparse signals that are based on efficient ℓ1 optimization methods. We first present an analysis for one simple algorithm (BPDN-DF) that works well when the system dynamics are known exactly. We then introduce a novel second algorithm (RWL1-DF) that is more computationally complex than BPDN-DF but performs better in practice, especially in the case where the system dynamics model is inaccurate. Robustness to model inaccuracy is achieved by using a hierarchical probabilistic data model and propagating higher-order statistics from the previous estimate (akin to Kalman filtering) in the sparse inference process. We demonstrate the properties of these algorithms on both simulated data as well as natural video sequences. Taken together, the algorithms presented in this paper represent the first strong performance analysis of dynamic filtering algorithms for time-varying sparse signals as well as state-of-the-art performance in this emerging application.

\ell _1

Sparse approximation is an optimization program that produces state-of-the-art results in many applications in signal processing and engineering. To deploy this approach in real-time, it is necessary to develop faster solvers than are currently available in digital. The Locally Competitive Algorithm (LCA) is a dynamical system designed to solve the class of sparse approximation problems in continuous time. But before implementing this network in analog VLSI, it is essential to provide performance guarantees. This paper presents new results on the convergence of the LCA neural network. Using recently-developed methods that make use of the Łojasiewicz inequality for nonsmooth functions, we prove that the output and state trajectories converge to a single fixed point. This improves on previous results by guaranteeing convergence to a singleton even when the optimization program has infinitely many and non-isolated solution points.

Minimization

Recovering static signals from compressed measurements is an important problem that has been extensively studied in modern signal processing. However, only recently have methods been proposed to tackle the problem of recovering a time-varying sequence from streaming online compressed measurements. In this paper, we study the capacity of the standard iterative soft-thresholding algorithm (ISTA) to perform this task in real-time. In previous work, ISTA has been shown to recover static sparse signals. The present paper demonstrates its ability to perform this recovery online in the dynamical setting where measurements are constantly streaming. Our analysis shows that the ℓ2-distance between the output and the target signal decays according to a linear rate, and is supported by simulations on synthetic and real data.

Convergence of a neural network for sparse approximation using the nonsmooth Łojasiewicz inequality

Analog circuits and systems research and education can benefit from the flexibility provided by large-scale Field Programmable Analog Arrays (FPAAs). This demonstration will present visitors with the hardware and software infrastructure supporting the use of a family of floating-gate based FPAAs being developed at Georgia Tech [1]. A picture of the programming and control hardware that will be demonstrated is found in Figure la. Figure lb shows the software flow that will be demonstrated [2]. The infrastructure is compact and portable and provides the user with a comprehensive set of tools for custom analog circuit design and implementation. The infrastructure includes the FPAA integrated circuit (IC); discrete analog to digital converters (ADC), digital to analog converters (DAC) and amplifier ICs; a 32-Bit ARM based microcontroller (μC) for interfacing the FPAA with a laptop computer; and Matlab and targeting software. The FPAA hardware communicates with Matlab over a Universal Serial Bus (USB) connection. The USB connection also provides the hardwares power. The software tools in the demonstration include three major systems: a Matlab Simulink FPAA program, a SPICE to FPAA compiler called GRASPER, and a visualization tool called RAT. Figure 2 shows a block diagram of the Programming and Control board and demonstration setup.

Iterative soft-thresholding for time-varying signal recovery

In this work, we present POUTINE, a novel estimator of the auto-correlation function (or more generally, the cross-correlation function) of ergodic stationary signals, an important task in a variety of applications. This estimator sparsely and non-adaptively samples the process via Bernoulli selection, generalizing the classical estimator in a natural way, and offering significant sampling reductions while sacrificing a modest degree of accuracy. Both the mean and variance of our estimator are explicitly analyzed, and in particular, we show that POUTINE gives an unbiased estimate of the classical estimator, which in turn gives an unbiased estimate of the underlying second-order statistics of interest. Furthermore, we show that POUTINE is a consistent estimator with variance approaching zero asymptotically. We demonstrate favorable performance of this approach for a simple stochastic process.