Stephen J. Wright

Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and Other Inverse Problems

Many problems in signal processing and statistical inference involve finding sparse solutions to under-determined, or ill-conditioned, linear systems of equations. A standard approach consists in minimizing an objective function which includes a quadratic (squared ) error term combined with a sparseness-inducing regularization term. Basis pursuit, the least absolute shrinkage and selection operator (LASSO), wavelet-based deconvolution, and compressed sensing are a few well-known examples of this approach. This paper proposes gradient projection (GP) algorithms for the bound-constrained quadratic programming (BCQP) formulation of these problems. We test variants of this approach that select the line search parameters in different ways, including techniques based on the Barzilai-Borwein method. Computational experiments show that these GP approaches perform well in a wide range of applications, often being significantly faster (in terms of computation time) than competing methods. Although the performance of GP methods tends to degrade as the regularization term is de-emphasized, we show how they can be embedded in a continuation scheme to recover their efficient practical performance.

Sparse Reconstruction by Separable Approximation

Finding sparse approximate solutions to large underdetermined linear systems of equations is a common problem in signal/image processing and statistics. Basis pursuit, the least absolute shrinkage and selection operator (LASSO), wavelet-based deconvolution and reconstruction, and compressed sensing (CS) are a few well-known areas in which problems of this type appear. One standard approach is to minimize an objective function that includes a quadratic (lscr 2) error term added to a sparsity-inducing (usually lscr1) regularizater. We present an algorithmic framework for the more general problem of minimizing the sum of a smooth convex function and a nonsmooth, possibly nonconvex regularizer. We propose iterative methods in which each step is obtained by solving an optimization subproblem involving a quadratic term with diagonal Hessian (i.e., separable in the unknowns) plus the original sparsity-inducing regularizer; our approach is suitable for cases in which this subproblem can be solved much more rapidly than the original problem. Under mild conditions (namely convexity of the regularizer), we prove convergence of the proposed iterative algorithm to a minimum of the objective function. In addition to solving the standard lscr2-lscr1 case, our framework yields efficient solution techniques for other regularizers, such as an lscrinfin norm and group-separable regularizers. It also generalizes immediately to the case in which the data is complex rather than real. Experiments with CS problems show that our approach is competitive with the fastest known methods for the standard lscr2-lscr1 problem, as well as being efficient on problems with other separable regularization terms.

Computational Methods for Sparse Solution of Linear Inverse Problems

The goal of the sparse approximation problem is to approximate a target signal using a linear combination of a few elementary signals drawn from a fixed collection. This paper surveys the major practical algorithms for sparse approximation. Specific attention is paid to computational issues, to the circumstances in which individual methods tend to perform well, and to the theoretical guarantees available. Many fundamental questions in electrical engineering, statistics, and applied mathematics can be posed as sparse approximation problems, making these algorithms versatile and relevant to a plethora of applications.

Power Awareness in Network Design and Routing

Exponential bandwidth scaling has been a fundamental driver of the growth and popularity of the Internet. However, increases in bandwidth have been accompanied by increases in power consumption, and despite sustained system design efforts to address power demand, significant technological challenges remain that threaten to slow future bandwidth growth. In this paper we describe the power and associated heat management challenges in todays routers. We advocate a broad approach to addressing this problem that includes making power-awareness a primary objective in the design and configuration of networks, and in the design and implementation of network protocols. We support our arguments by providing a case study of power demands of two standard router platforms that enables us to create a generic model for router power consumption. We apply this model in a set of target network configurations and use mixed integer optimization techniques to investigate power consumption, performance and robustness in static network design and in dynamic routing. Our results indicate the potential for significant power savings in operational networks by including power-awareness.

Distributed MPC Strategies With Application to Power System Automatic Generation Control

A distributed model predictive control (MPC) framework, suitable for controlling large-scale networked systems such as power systems, is presented. The overall system is decomposed into subsystems, each with its own MPC controller. These subsystem-based MPCs work iteratively and cooperatively towards satisfying systemwide control objectives. If available computational time allows convergence, the proposed distributed MPC framework achieves performance equivalent to centralized MPC. Furthermore, the distributed MPC algorithm is feasible and closed-loop stable under intermediate termination. Automatic generation control (AGC) provides a practical example for illustrating the efficacy of the proposed distributed MPC framework.

Application of interior-point methods to model predictive control

We present a structured interior-point method for the efficient solution of the optimal control problem in model predictive control. The cost of this approach is linear in the horizon length, compared with cubic growth for a naive approach. We use a discrete-time Riccati recursion to solve the linear equations efficiently at each iteration of the interior-point method, and show that this recursion is numerically stable. We demonstrate the effectiveness of the approach by applying it to three process control problems.

Nonlinear Predictive Control and Moving Horizon Estimation — An Introductory Overview

In the past decade model predictive control (MPC) has become a preferred control strategy for a large number of processes. The main reasons for this preference include the ability to handle constraints in an optimal way and the flexible formulation in the time domain. Linear MPC schemes, i.e. MPC schemes for which the prediction is based on a linear description of the plant, are by now routinely used in a number of industrial sectors and the underlying control theoretic problems, like stability, are well studied. Nonlinear model predictive control (NMPC), i.e. MPC based on a nonlinear plant description, has only emerged in the past decade and the number of reported industrial applications is still fairly low. Because of its additional ability to take process nonlinearities into account, expectations on this control methodology are high.

The empirical behavior of sampling methods for stochastic programming

We investigate the quality of solutions obtained from sample-average approximations to two-stage stochastic linear programs with recourse. We use a recently developed software tool executing on a computational grid to solve many large instances of these problems, allowing us to obtain high-quality solutions and to verify optimality and near-optimality of the computed solutions in various ways.

Cooperative distributed model predictive control

Abstract In this paper we propose a cooperative distributed linear model predictive control strategy applicable to any finite number of subsystems satisfying a stabilizability condition. The control strategy has the following features: hard input constraints are satisfied; terminating the iteration of the distributed controllers prior to convergence retains closed-loop stability; in the limit of iterating to convergence, the control feedback is plantwide Pareto optimal and equivalent to the centralized control solution; no coordination layer is employed. We provide guidance in how to partition the subsystems within the plant. We first establish exponential stability of suboptimal model predictive control and show that the proposed cooperative control strategy is in this class. We also establish that under perturbation from a stable state estimator, the origin remains exponentially stable. For plants with sparsely coupled input constraints, we provide an extension in which the decision variable space of each suboptimization is augmented to achieve Pareto optimality. We conclude with a simple example showing the performance advantage of cooperative control compared to noncooperative and decentralized control strategies.

Simultaneous Variable Selection

We propose a new method for selecting a common subset of explanatory variables where the aim is to model several response variables. The idea is a natural extension of the LASSO technique proposed by Tibshirani (1996) and is based on the (joint) residual sum of squares while constraining the parameter estimates to lie within a suitable polyhedral region. The properties of the resulting convex programming problem are analyzed for the special case of an orthonormal design. For the general case, we develop an efficient interior point algorithm. The method is illustrated on a dataset with infrared spectrometry measurements on 14 qualitatively different but correlated responses using 770 wavelengths. The aim is to select a subset of the wavelengths suitable for use as predictors for as many of the responses as possible.