Damek Davis

Convergence Rate Analysis of Several Splitting Schemes

Operator-splitting schemes are iterative algorithms for solving many types of numerical problems. A lot is known about these methods: they converge, and in many cases we know how quickly they converge. But when they are applied to optimization problems, there is a gap in our understanding: The theoretical speed of operator-splitting schemes is nearly always measured in the ergodic sense, but ergodic operator-splitting schemes are rarely used in practice. In this chapter, we tackle the discrepancy between theory and practice and uncover fundamental limits of a class of operator-splitting schemes. Our surprising conclusion is that the relaxed Peaceman-Rachford splitting algorithm, a version of the Alternating Direction Method of Multipliers (ADMM), is nearly as fast as the proximal point algorithm in the ergodic sense and nearly as slow as the subgradient method in the nonergodic sense. A large class of operator-splitting schemes extend from the relaxed Peaceman-Rachford splitting algorithm. Our results show that this class of operator-splitting schemes is also nearly as slow as the subgradient method. The tools we create in this chapter can also be used to prove nonergodic convergence rates of more general splitting schemes, so they are interesting in their own right.

A Three-Operator Splitting Scheme and its Optimization Applications

Operator-splitting methods convert optimization and inclusion problems into fixed-point equations; when applied to convex optimization and monotone inclusion problems, the equations given by operator-splitting methods are often easy to solve by standard techniques. The hard part of this conversion, then, is to design nicely behaved fixed-point equations. In this paper, we design a new, and thus far, the only nicely behaved fixed-point equation for solving monotone inclusions with three operators; the equation employs resolvent and forward operators, one at a time, in succession. We show that our new equation extends the Douglas-Rachford and forward-backward equations; we prove that standard methods for solving the equation converge; and we give two accelerated methods for solving the equation.

Faster Convergence Rates of Relaxed Peaceman-Rachford and ADMM Under Regularity Assumptions

In this paper, we provide a comprehensive convergence rate analysis of the Douglas-Rachford splitting (DRS), Peaceman-Rachford splitting (PRS), and alternating direction method of multipliers (ADMM) algorithms under various regularity assumptions including strong convexity, Lipschitz differentiability, and bounded linear regularity. The main consequence of this work is that relaxed PRS and ADMM automatically adapt to the regularity of the problem and achieve convergence rates that improve upon the (tight) worst-case rates that hold in the absence of such regularity. All of the results are obtained using simple techniques.

Convergence Rate Analysis of Primal-Dual Splitting Schemes

Primal-dual splitting schemes are a class of powerful algorithms that solve complicated monotone inclusions and convex optimization problems that are built from many simpler pieces. They decompose problems that are built from sums, linear compositions, and infimal convolutions of simple functions so that each simple term is processed individually via proximal mappings, gradient mappings, and multiplications by the linear maps. This leads to easily implementable and highly parallelizable or distributed algorithms, which often obtain nearly state-of-the-art performance. In this paper, we analyze a monotone inclusion problem that captures a large class of primal-dual splittings as a special case. We introduce a unifying scheme and use some abstract analysis of the algorithm to prove convergence rates of the proximal point algorithm, forward-backward splitting, Peaceman--Rachford splitting, and forward-backward-forward splitting applied to the model problem. Our ergodic convergence rates are deduced under var...

Convergence Rate Analysis of the Forward-Douglas-Rachford Splitting Scheme

Operator splitting schemes are a class of powerful algorithms that solve complicated monotone inclusion and convex optimization problems that are built from many simpler pieces. They give rise to algorithms in which all simple pieces of the decomposition are processed individually. This leads to easily implementable and highly parallelizable or distributed algorithms, which often obtain nearly state-of-the-art performance. In this paper, we analyze the convergence rate of the forward-Douglas--Rachford splitting (FDRS) algorithm, which is a generalization of the forward-backward splitting (FBS) and Douglas--Rachford splitting algorithms. Under general convexity assumptions, we derive the ergodic and nonergodic convergence rates of the FDRS algorithm, and show that these rates are the best possible. Under Lipschitz differentiability assumptions, we show that the best iterate of FDRS converges as quickly as the last iterate of the FBS algorithm. Under strong convexity assumptions, we derive convergence rates...

Tactical Scheduling for Precision Air Traffic Operations: Past Research and Current Problems

Future air transportation systems stand to benefit significantly in safety and efficiency from the predictable movement of aircraft along precisely defined paths in the airspace. Such aircraft movement, hereafter referred to as Precision Air Traffic Operations (PATO), is not widely used during periods of peak air traffic in today’s system, but is the foundation of high-capacity operations envisioned for the future. Automation and deployment of PATO, being a relatively young field of research, has not had time to establish structured theories and standardized reference literature. As a consequence, researchers interested in entering this field have difficulty applying to it classical techniques of operations research and optimal control theory. The main obstacle to such applications is the lack of access to the finer domain knowledge of air traffic operations (most importantly, knowledge of the operational constraints) needed to formulate research problems that promise deployable automation tools. Such a formulation requires the researcher to characterize and assess the research efforts and tendencies that emerged in the recent decades to address diverse problems, some outwardly similar yet essentially different. The research field of PATO has now matured to a stage where this requirement can start being met. This paper, aimed as a step in this direction, provides (a) a formulation of the general problem of defining conceptually, constructing, and using a schedule for PATO that contains specification of merging sequences and provides aircraft separation continuously in time, (b) the context necessary for understanding the formulation and its limitations, and (c) a review of prior research on future Air Traffic Operations (ATO) and, in particular, on the role of Air Traffic Control (ATC) in these operations.

Multi-view feature engineering and learning

We frame the problem of local representation of imaging data as the computation of minimal sufficient statistics that are invariant to nuisance variability induced by viewpoint and illumination. We show that, under very stringent conditions, these are related to “feature descriptors” commonly used in Computer Vision. Such conditions can be relaxed if multiple views of the same scene are available. We propose a sampling-based and a point-estimate based approximation of such a representation, compared empirically on image-to-(multiple)image matching, for which we introduce a multi-view wide-baseline matching benchmark, consisting of a mixture of real and synthetic objects with ground truth camera motion and dense three-dimensional geometry.

Efficient Computation of Separation-Compliant Speed Advisories for Air Traffic Arriving in Terminal Airspace

A class of problems in air traffic management asks for a scheduling algorithm that supplies the air traffic services authority not only with a schedule of arrivals and departures, but also with speed advisories. Since advisories must be finite, a scheduling algorithm must ultimately produce a finite data set, hence must either start with a purely discrete model or involve a discretization of a continuous one. The former choice, often preferred for intuitive clarity, naturally leads to mixed-integer programs, hindering proofs of correctness and computational cost bounds (crucial for real-time operations). In this paper, a hybrid control system is used to model air traffic scheduling, capturing both the discrete and continuous aspects. This framework is applied to a class of problems, called the Fully Routed Nominal Problem. We prove a number of geometric results on feasible schedules and use these results to formulate an algorithm that attempts to compute a collective speed advisory, effectively finite, and has computational cost polynomial in the number of aircraft. This work is a first step toward optimization and models refined with more realistic detail.

Asymmetric Sparse Kernel Approximations for Large-Scale Visual Search

We introduce an asymmetric sparse approximate embedding optimized for fast kernel comparison operations arising in large-scale visual search. In contrast to other methods that perform an explicit approximate embedding using kernel PCA followed by a distance compression technique in Rd, which loses information at both steps, our method utilizes the implicit kernel representation directly. In addition, we empirically demonstrate that our method needs no explicit training step and can operate with a dictionary of random exemplars from the dataset. We evaluate our method on three benchmark image retrieval datasets: SIFT1M, ImageNet, and 80M-TinyImages.

Factorial and Noetherian subrings of power series rings

Let F be a field. We show that certain subrings contained between the polynomial ring F[X] = F[X1, ⋯ ,Xn] and the power series ring F[X][[Y ]] = F[X1, ⋯ ,Xn][[Y ]] have Weierstrass Factorization, which allows us to deduce both unique factorization and the Noetherian property. These intermediate subrings are obtained from elements of F[X][[Y ]] by bounding their total X-degree above by a positive real-valued monotonic up function λ on their Y -degree. These rings arise naturally in studying the p-adic analytic variation of zeta functions over finite fields. Future research into this area may study more complicated subrings in which Y = (Y1, ⋯ , Ym) has more than one variable, and for which there are multiple degree functions, λ1, ⋯ , λm. Another direction of study would be to generalize these results to k-affinoid algebras.