Alexey F. Izmailov

Newton-type methods for optimization and variational problems

1. Elements of optimization theory and variational analysis.- 2. Equations and unconstrained optimization.- 3. Variational problems: local methods.- 4. Constrained optimization: local methods.- 5. Variational problems: globalization of convergence.- 6. Constrained optimization: globalization of convergence.- 7. Degenerate problems with non-isolated solutions.- A. Miscellaneous material.

Stabilized SQP revisited

The stabilized version of the sequential quadratic programming algorithm (sSQP) had been developed in order to achieve superlinear convergence in situations when the Lagrange multipliers associated to a solution are not unique. Within the framework of Fischer (Math Program 94:91–124, 2002), the key to local superlinear convergence of sSQP are the following two properties: upper Lipschitzian behavior of solutions of the Karush-Kuhn-Tucker (KKT) system under canonical perturbations and local solvability of sSQP subproblems with the associated primal-dual step being of the order of the distance from the current iterate to the solution set of the unperturbed KKT system. According to Fernández and Solodov (Math Program 125:47–73, 2010), both of these properties are ensured by the second-order sufficient optimality condition (SOSC) without any constraint qualification assumptions. In this paper, we state precise relationships between the upper Lipschitzian property of solutions of KKT systems, error bounds for KKT systems, the notion of critical Lagrange multipliers (a subclass of multipliers that violate SOSC in a very special way), the second-order necessary condition for optimality, and solvability of sSQP subproblems. Moreover, for the problem with equality constraints only, we prove superlinear convergence of sSQP under the assumption that the dual starting point is close to a noncritical multiplier. Since noncritical multipliers include all those satisfying SOSC but are not limited to them, we believe this gives the first superlinear convergence result for any Newtonian method for constrained optimization under assumptions that do not include any constraint qualifications and are weaker than SOSC. In the general case when inequality constraints are present, we show that such a relaxation of assumptions is not possible. We also consider applying sSQP to the problem where inequality constraints are reformulated into equalities using slack variables, and discuss the assumptions needed for convergence in this approach. We conclude with consequences for local regularization methods proposed in (Izmailov and Solodov SIAM J Optim 16:210–228, 2004; Wright SIAM J. Optim. 15:673–676, 2005). In particular, we show that these methods are still locally superlinearly convergent under the noncritical multiplier assumption, weaker than SOSC employed originally.

Newton-Type Methods for Optimization Problems without Constraint Qualifications

We consider equality-constrained optimization problems, where a given solution may not satisfy any constraint qualification but satisfies the standard second-order sufficient condition for optimality. Based on local identification of the rank of the constraints degeneracy via the singular-value decomposition, we derive a modified primal-dual optimality system whose solution is locally unique, nondegenerate, and thus can be found by standard Newton-type techniques. Using identification of active constraints, we further extend our approach to mixed equality- and inequality-constrained problems, and to mathematical programs with complementarity constraints (MPCC). In particular, for MPCC we obtain a local algorithm with quadratic convergence under the second-order sufficient condition only, without any constraint qualifications, not even the special MPCC constraint qualifications.

The Theory of 2-Regularity for Mappings with Lipschitzian Derivatives and its Applications to Optimality Conditions

We study local structure of a nonlinear mapping near points where standard regularity and/or smoothness assumptions need not be satisfied. We introduce a new concept of 2-regularity a certain kind of second-order regularity for a once differentiable mapping whose derivative is Lipschitz continuous. Under this 2-regularity condition, we obtain the representation theorem and the covering theorem i.e., stability with respect to “right-hand side” perturbations under assumptions that are weaker than those previously employed in the literature for results of this type. These results are further used to derive a constructive description of the tangent cone to a set defined by 2-regular equality constraints and optimality conditions for related optimization problems. The class of mappings introduced and studied in the paper appears to be a convenient tool for treating complementarity structures by means of an appropriate equation-based reformulation. Optimality conditions for mathematical programs with equivalently reformulated complementarity constraints are also discussed.

On attraction of Newton-type iterates to multipliers violating second-order sufficiency conditions

Assuming that the primal part of the sequence generated by a Newton-type (e.g., SQP) method applied to an equality-constrained problem converges to a solution where the constraints are degenerate, we investigate whether the dual part of the sequence is attracted by those Lagrange multipliers which satisfy second-order sufficient condition (SOSC) for optimality, or by those multipliers which violate it. This question is relevant at least for two reasons: one is speed of convergence of standard methods; the other is applicability of some recently proposed approaches for handling degenerate constraints. We show that for the class of damped Newton methods, convergence of the dual sequence to multipliers satisfying SOSC is unlikely to occur. We support our findings by numerical experiments. We also suggest a simple auxiliary procedure for computing multiplier estimates, which does not have this undesirable property. Finally, some consequences for the case of mixed equality and inequality constraints are discussed.

An Active-Set Newton Method for Mathematical Programs with Complementarity Constraints

For a mathematical program with complementarity constraints (MPCC), we propose an active-set Newton method, which has the property of local quadratic convergence under the MPCC linear independence constraint qualification (MPCC-LICQ) and the standard second-order sufficient condition (SOSC) for optimality. Under MPCC-LICQ, this SOSC is equivalent to the piecewise SOSC on branches of MPCC, which is weaker than the special MPCC-SOSC often employed in the literature. The piecewise SOSC is also more natural than MPCC-SOSC because, unlike the latter, it has an appropriate second-order necessary condition as its counterpart. In particular, our assumptions for local quadratic convergence are weaker than those required by standard SQP when applied to MPCC and are equivalent to assumptions required by piecewise SQP for MPCC. Moreover, each iteration of our method consists of solving a linear system of equations instead of a quadratic program. Some globalization issues of the local scheme are also discussed, and illustrative examples and numerical experiments are presented.

A Class of Active-Set Newton Methods for Mixed Complementarity Problems

Based on the identification of indices active at a solution of the mixed complementarity problem (MCP), we propose a class of Newton methods for which local superlinear convergence holds under extremely mild assumptions. In particular, the error bound condition needed for the identification procedure and the nondegeneracy condition needed for the convergence of the resulting Newton method are individually and collectively strictly weaker than the property of semistability of a solution. Thus the local superlinear convergence conditions of the presented method are weaker than conditions required for the semismooth (generalized) Newton methods applied to MCP reformulations. Moreover, they are also weaker than convergence conditions of the linearization (Josephy--Newton) method. For the special case of optimality systems with primal-dual structure, we further consider the question of superlinear convergence of primal variables. We illustrate our theoretical results with numerical experiments on some specially constructed MCPs whose solutions do not satisfy the usual regularity assumptions.

Optimality Conditions for Irregular Inequality-Constrained Problems

We consider feasible sets given by conic constraints, where the cone defining the constraints is convex with nonempty interior. We study the case where the feasible set is not assumed to be regular in the classical sense of Robinson and obtain a constructive description of the tangent cone under a certain new second-order regularity condition. This condition contains classical regularity as a special case, while being weaker when constraints are twice differentiable. Assuming that the cone defining the constraints is finitely generated, we also derive a special form of primal-dual optimality conditions for the corresponding constrained optimization problem. Our results subsume optimality conditions for both the classical regular and second-order regular cases, while still being meaningful in the more general setting in the sense that the multiplier associated with the objective function is nonzero.

Karush-Kuhn-Tucker systems: regularity conditions, error bounds and a class of Newton-type methods

Abstract. We consider optimality systems of Karush-Kuhn-Tucker (KKT) type, which arise, for example, as primal-dual conditions characterizing solutions of optimization problems or variational inequalities. In particular, we discuss error bounds and Newton-type methods for such systems. An exhaustive comparison of various regularity conditions which arise in this context is given. We obtain a new error bound under an assumption which we show to be strictly weaker than assumptions previously used for KKT systems, such as quasi-regularity or semistability (equivalently, the R0-property). Error bounds are useful, among other things, for identifying active constraints and developing efficient local algorithms. We propose a family of local Newton-type algorithms. This family contains some known active-set Newton methods, as well as some new methods. Regularity conditions required for local superlinear convergence compare favorably with convergence conditions of nonsmooth Newton methods and sequential quadratic programming methods.

Global Convergence of Augmented Lagrangian Methods Applied to Optimization Problems with Degenerate Constraints, Including Problems with Complementarity Constraints

We consider global convergence properties of the augmented Lagrangian methods on problems with degenerate constraints, with a special emphasis on mathematical programs with complementarity constraints (MPCC). In the general case, we show convergence to stationary points of the problem under an error bound condition for the feasible set (which is weaker than constraint qualifications), assuming that the iterates have some modest features of approximate local minimizers of the augmented Lagrangian. For MPCC, we first argue that even weak forms of general constraint qualifications that are suitable for convergence of the augmented Lagrangian methods, such as the recently proposed relaxed positive linear dependence condition, should not be expected to hold and thus special analysis is needed. We next obtain a rather complete picture, showing that, under this contexts usual MPCC-linear independence constraint qualification, feasible accumulation points of the iterates are guaranteed to be C-stationary for MPC...