François Glineur

First-order methods of smooth convex optimization with inexact oracle

We introduce the notion of inexact first-order oracle and analyze the behavior of several first-order methods of smooth convex optimization used with such an oracle. This notion of inexact oracle naturally appears in the context of smoothing techniques, Moreau–Yosida regularization, Augmented Lagrangians and many other situations. We derive complexity estimates for primal, dual and fast gradient methods, and study in particular their dependence on the accuracy of the oracle and the desired accuracy of the objective function. We observe that the superiority of fast gradient methods over the classical ones is no longer absolute when an inexact oracle is used. We prove that, contrary to simple gradient schemes, fast gradient methods must necessarily suffer from error accumulation. Finally, we show that the notion of inexact oracle allows the application of first-order methods of smooth convex optimization to solve non-smooth or weakly smooth convex problems.

Low-Rank Matrix Approximation with Weights or Missing Data Is NP-Hard

Weighted low-rank approximation (WLRA), a dimensionality reduction technique for data analysis, has been successfully used in several applications, such as in collaborative filtering to design recommender systems or in computer vision to recover structure from motion. In this paper, we prove that computing an optimal WLRA is NP-hard, already when a rank-one approximation is sought. In fact, we show that it is hard to compute approximate solutions to the WLRA problem with some prescribed accuracy. Our proofs are based on reductions from the maximum-edge biclique problem and apply to strictly positive weights as well as to binary weights (the latter corresponding to low-rank matrix approximation with missing data).

Recent Advances in Optimization and its Applications in Engineering

Mathematical optimization encompasses both a rich and rapidly evolving body of fundamental theory, and a variety of exciting applications in science and engineering. The present book contains a careful selection of articles on recent advances in optimization theory, numerical methods, and their applications in engineering. It features in particular new methods and applications in the fields of optimal control, PDE-constrained optimization, nonlinear optimization, and convex optimization. The authors of this volume took part in the 14th Belgian-French-German Conference on Optimization (BFG09) organized in Leuven, Belgium, on September 14-18, 2009. The volume contains a selection of reviewed articles contributed by the conference speakers as well as three survey articles by plenary speakers and two papers authored by the winners of the best talk and best poster prizes awarded at BFG09. Researchers and graduate students in applied mathematics, computer science, and many branches of engineering will find in this book an interesting and useful collection of recent ideas on the methods and applications of optimization.

Two algorithms for orthogonal nonnegative matrix factorization with application to clustering

Abstract Approximate matrix factorization techniques with both nonnegativity and orthogonality constraints, referred to as orthogonal nonnegative matrix factorization (ONMF), have been recently introduced and shown to work remarkably well for clustering tasks such as document classification. In this paper, we introduce two new methods to solve ONMF. First, we show mathematical equivalence between ONMF and a weighted variant of spherical k-means, from which we derive our first method, a simple EM-like algorithm. This also allows us to determine when ONMF should be preferred to k-means and spherical k-means. Our second method is based on an augmented Lagrangian approach. Standard ONMF algorithms typically enforce nonnegativity for their iterates while trying to achieve orthogonality at the limit (e.g., using a proper penalization term or a suitably chosen search direction). Our method works the opposite way: orthogonality is strictly imposed at each step while nonnegativity is asymptotically obtained, using a quadratic penalty. Finally, we show that the two proposed approaches compare favorably with standard ONMF algorithms on synthetic, text and image data sets.

Linear convergence of first order methods for non-strongly convex optimization

The standard assumption for proving linear convergence of first order methods for smooth convex optimization is the strong convexity of the objective function, an assumption which does not hold for many practical applications. In this paper, we derive linear convergence rates of several first order methods for solving smooth non-strongly convex constrained optimization problems, i.e. involving an objective function with a Lipschitz continuous gradient that satisfies some relaxed strong convexity condition. In particular, in the case of smooth constrained convex optimization, we provide several relaxations of the strong convexity conditions and prove that they are sufficient for getting linear convergence for several first order methods such as projected gradient, fast gradient and feasible descent methods. We also provide examples of functional classes that satisfy our proposed relaxations of strong convexity conditions. Finally, we show that the proposed relaxed strong convexity conditions cover important applications ranging from solving linear systems, Linear Programming, and dual formulations of linearly constrained convex problems.

Document classification using nonnegative matrix factorization and underapproximation

In this study, we use nonnegative matrix factorization (NMF) and nonnegative matrix underapproximation (NMU) approaches to generate feature vectors that can be used to cluster Aviation Safety Reporting System (ASRS) documents obtained from the Distributed National ASAP Archive (DNAA). By preserving nonnegativity, both the NMF and NMU facilitate a sum-of-parts representation of the underlying term usage patterns in the ASRS document collection. Both the training and test sets of ASRS documents are parsed and then factored by both algorithms to produce a reduced-rank representations of the entire document space. The resulting feature and coefficient matrix factors are used to cluster ASRS documents so that the (known) associated anomalies of training documents are directly mapped to the feature vectors. Dominant features of test documents are then used to generate anomaly relevance scores for those documents.We demonstrate that the approximate solution obtained by NMU using Lagrangrian duality can lead to a better sum-of-parts representation and document classification accuracy.

A Global-Local Synthesis Approach for Large Non-Regular Arrays

A method is proposed for the synthesis of large planar non-regular arrays assuming constant magnitude for the excitation of the elements. The approach, which combines global and local optimization, is based on distinguishing aperture-type and non-coherent parts of the array factor. The following constraints are considered: minimal separation of the antennas, maximum size of the array and fixed mainbeam width. The level of the sidelobes is reduced via the minimization of a new type of flexible averaging cost function based on an Lp-norm. Possible applications of this model lie in the field of radio astronomy, satellite communications and radar systems. The proposed optimization strategy consists of three successive steps designed to be independent of each other. Starting with an equivalent continuous aperture, the first two steps act as global transformations of the radial and azimuthal positions of the elements in the initial array, while the third step performs a local optimization of individual elements of the array. This last step heavily relies on computations of the gradient of the cost function, which can be done quickly using an FFT-based procedure. The method is illustrated for several large-scale examples by considering as inputs four different types of non-regular arrays.

Heuristics for exact nonnegative matrix factorization

The exact nonnegative matrix factorization (exact NMF) problem is the following: given an m-by-n nonnegative matrix X and a factorization rank r, find, if possible, an m-by-r nonnegative matrix W and an r-by-n nonnegative matrix H such that

Proving strong duality for geometric optimization using a conic formulation

X = WH