Robert M. Gower

Randomized Iterative Methods for Linear Systems

We develop a novel, fundamental and surprisingly simple randomized iterative method for solving consistent linear systems. Our method has six different but equivalent interpretations: sketch-and-project, constrain-and-approximate, random intersect, random linear solve, random update and random fixed point. By varying its two parameters

Randomized Quasi-Newton Updates Are Linearly Convergent Matrix Inversion Algorithms

-

Computing the sparsity pattern of Hessians using automatic differentiation

a positive definite matrix (defining geometry), and a random matrix (sampled in an independently and identically distributed fashion in each iteration)

Higher-order reverse automatic differentiation with emphasis on the third-order

-

Characterising particulate random media from near-surface backscattering: A machine learning approach to predict particle size and concentration

we recover a comprehensive array of well-known algorithms as special cases, including the randomized Kaczmarz method, randomized Newton method, randomized coordinate descent method and random Gaussian pursuit. We naturally also obtain variants of all these methods using blocks and importance sampling. However, our method allows for a much wider selection of these two parameters, which leads to a number of new specific methods. We prove exponential convergence of the expected norm of the error in a single theorem, from which existing complexity results for known variants can be obtained. However, we also give an exact formula for the evolution of the expected iterates, which allows us to give lower bounds on the convergence rate.

Characterizing composites with acoustic backscattering: Combining data driven and analytical methods

We develop and analyze a broad family of stochastic/randomized algorithms for calculating an approximate inverse matrix. We also develop specialized variants maintaining symmetry or positive definiteness of the iterates. All methods in the family converge globally and linearly (i.e., the error decays exponentially), with explicit rates. In special cases, we obtain stochastic block variants of several quasi-Newton updates, including bad Broyden (BB), good Broyden (GB), Powell-symmetric-Broyden (PSB), Davidon--Fletcher--Powell (DFP), and Broyden--Fletcher--Goldfarb--Shanno (BFGS). Ours are the first stochastic versions of these updates shown to converge to an inverse of a fixed matrix. Through a dual viewpoint we uncover a fundamental link between quasi-Newton updates and approximate inverse preconditioning. Further, we develop an adaptive variant of randomized block BFGS, where we modify the distribution underlying the stochasticity of the method throughout the iterative process to achieve faster convergen...

Stochastic block BFGS: squeezing more curvature out of data

We compare two methods that calculate the sparsity pattern of Hessian matrices using the computational framework of automatic differentiation. The first method is a forward-mode algorithm by Andrea Walther in 2008 which has been implemented as the driver called hess_pat in the automatic differentiation package ADOL-C. The second is edge_push_sp, a new reverse mode algorithm descended from the edge_pushing algorithm for calculating Hessians by Gower and Mello in 2012. We present complexity analysis and perform numerical tests for both algorithms. The results show that the new reverse algorithm is very promising.

Stochastic Dual Ascent for Solving Linear Systems.

It is commonly assumed that calculating third order information is too expensive for most applications. But we show that the directional derivative of the Hessian (

Linearly Convergent Randomized Iterative Methods for Computing the Pseudoinverse

D^3 f(x)\cdot d