Aaron Sidford

Efficient Accelerated Coordinate Descent Methods and Faster Algorithms for Solving Linear Systems

In this paper we show how to accelerate randomized coordinate descent methods and achieve faster convergence rates without paying per-iteration costs in asymptotic running time. In particular, we show how to generalize and efficiently implement a method proposed by Nesterov, giving faster asymptotic running times for various algorithms that use standard coordinate descent as a black box. In addition to providing a proof of convergence for this new general method, we show that it is numerically stable, efficiently implementable, and in certain regimes, asymptotically optimal. To highlight the power of this algorithm, we show how it can used to create faster linear system solvers in several regimes: - We show how this method achieves a faster asymptotic runtime than conjugate gradient for solving a broad class of symmetric positive definite systems of equations. - We improve the convergence guarantees for Kaczmarz methods, a popular technique for image reconstruction and solving over determined systems of equations, by accelerating an algorithm of Strohmer and Vershynin. - We achieve the best known running time for solving Symmetric Diagonally Dominant (SDD) system of equations in the unit-cost RAM model, obtaining a running time of O(m log3/2n (log log n)1/2 log((log n)/eps)) by accelerating a recent solver by Kelner et al. Beyond the independent interest of these solvers, we believe they highlight the versatility of the approach of this paper and we hope that they will open the door for further algorithmic improvements in the future.

Uniform Sampling for Matrix Approximation

Random sampling has become a critical tool in solving massive matrix problems. For linear regression, a small, manageable set of data rows can be randomly selected to approximate a tall, skinny data matrix, improving processing time significantly. For theoretical performance guarantees, each row must be sampled with probability proportional to its statistical leverage score. Unfortunately, leverage scores are difficult to compute. A simple alternative is to sample rows uniformly at random. While this often works, uniform sampling will eliminate critical row information for many natural instances. We take a fresh look at uniform sampling by examining what information it does preserve. Specifically, we show that uniform sampling yields a matrix that, in some sense, well approximates a large fraction of the original. While this weak form of approximation is not enough for solving linear regression directly, it is enough to compute a better approximation. This observation leads to simple iterative row sampling algorithms for matrix approximation that run in input-sparsity time and preserve row structure and sparsity at all intermediate steps. In addition to an improved understanding of uniform sampling, our main proof introduces a structural result of independent interest: we show that every matrix can be made to have low coherence by reweighting a small subset of its rows.

A Faster Cutting Plane Method and its Implications for Combinatorial and Convex Optimization

In this paper we improve upon the running time for finding a point in a convex set given a separation oracle. In particular, given a separation oracle for a convex set K ⊂ Rⁿ that is contained in a box of radius R we show how to either compute a point in K or prove that K does not contain a ball of radius ϵ using an expected O(n log(nR/ϵ)) evaluations of the oracle and additional time O(n³ log^O(1)(nR/ϵ)). This matches the oracle complexity and improves upon the O(n^ω+1 log(nR/ϵ)) additional time of the previous fastest algorithm achieved over 25 years ago by Vaidya [91] for the current value of the matrix multiplication constant w <; 2.373 [98], [36] when R/ϵ = O(poly(n)). Using a mix of standard reductions and new techniques we show how our algorithm can be used to improve the running time for solving classic problems in continuous and combinatorial optimization. In particular we provide the following running time improvements: · Submodular Function Minimization: n is the size of the ground set, M is the maximum absolute value of function values and EO is the time for function evaluation. Our weakly and strongly polynomial time algorithms have a running time of O(n² log nM · EO + n³ log^O(1) nM) and O(n³ log² n · EO + n⁴ log^O(1) n), improving upon the previous best of O((n⁴ · EO + n⁵)logM) and O(n⁵ · EO + n⁶) respectively. · Submodular Flow: n = |V|, m = |E|, C is the maximum edge cost in absolute value and U is maximum edge capacity in absolute value. We obtain a faster weakly polynomial running time of O(n² log nCU · EO + n³ logO(1) nCU), improving upon the previous best of O(mn⁵ log nU · EO) and O (n⁴h min {log C, log U}) from 15 years ago by a factor of Õ(n⁴). We also achieve faster strongly polynomial time algorithms as a consequence of our result on submodular minimization. · Matroid Intersection: n is the size of the ground set, r is the maximum size of independent sets, M is the maximum absolute value of element weight, T_rank and T_ind are the time for each rank and independence oracle query. We obtain a running time of O((nr log² nT_rank+n³ log^O(1) n) log nM) and O((n² log nT_ind+n³ log^O(1) n) log nM), achieving the first quadratic bound on the query complexity for the independence and rank oracles. In the unweighted case, this is the first improvement since 1986 for independence oracle. · Semidefinite Programming: n is the number of constraints, m is the number of dimensions and S is the total number of non-zeros in the constraint matrices. We obtain a running time of O(n(n² + m^ω + S)), improving upon the previous best of Õ(n(n^ω + m^ω + S)) for the regime S is small.

Accelerated Methods for NonConvex Optimization

We present an accelerated gradient method for nonconvex optimization problems with Lipschitz continuous first and second derivatives. In a time

Single Pass Spectral Sparsification in Dynamic Streams

O(\epsilon^{-7/4} \log(1/ \epsilon) )

Geometric median in nearly linear time

, the method finds an

Efficient Inverse Maintenance and Faster Algorithms for Linear Programming

\epsilon

Spectrum Approximation Beyond Fast Matrix Multiplication: Algorithms and Hardness

-stationary point, meaning a point

Subquadratic submodular function minimization

x

Routing under balance

such that