Mathematics

This paper gives an in-depth review of the most common iterative methods for unconstrained optimization using two functions that belong to a class of Rosenbrock functions as a performance test. This study covers the Steepest Gradient Descent Method, the Newton-Raphson Method, and the Fletcher-Reeves Conjugate Gradient method. In addition, four different step-size selecting methods including fixed-step-size, variable step-size, quadratic-fit, and golden section method were considered. Due to the computational nature of solving minimization problems, testing the algorithms is an essential part of this paper. Therefore, an extensive set of numerical test results is also provided to present an insightful and a comprehensive comparison of the reviewed algorithms. This study highlights the differences and the trade-offs involved in comparing these algorithms.

Composite minimization is a powerful framework in large-scale convex optimization, based on decoupling of the objective function into terms with structurally different properties and allowing for more flexible algorithmic design. In this work, we introduce a new algorithmic framework for complementary composite minimization, where the objective function decouples into a (weakly) smooth and a uniformly convex term. This particular form of decoupling is pervasive in statistics and machine learning, due to its link to regularization. The main contributions of our work are summarized as follows. First, we introduce the problem of complementary composite minimization in general normed spaces; second, we provide a unified accelerated algorithmic framework to address broad classes of complementary composite minimization problems; and third, we prove that the algorithms resulting from our framework are near-optimal in most of the standard optimization settings. Additionally, we show that our algorithmic framework can be used to address the problem of making the gradients small in general normed spaces. As a concrete example, we obtain a nearly-optimal method for the standard ??1 setup (small gradients in the ????norm), essentially matching the bound of Nesterov (2012) that was previously known only for the Euclidean setup. Finally, we show that our composite methods are broadly applicable to a number of regression problems, leading to complexity bounds that are either new or match the best existing ones.

The Frank-Wolfe algorithm is a method for constrained optimization that relies on linear minimizations, as opposed to projections. Therefore, a motivation put forward in a large body of work on the Frank-Wolfe algorithm is the computational advantage of solving linear minimizations instead of projections. However, the discussions supporting this advantage are often too succinct or incomplete. In this paper, we review the complexity bounds for both tasks on several sets commonly used in optimization. Projection methods onto the ??p -ball, p?�]1,2[?�]2,+?�[ , and the Birkhoff polytope are also proposed.

In this paper, we consider a composite optimization problem with linear coupling constraints in a multi-agent network. In this problem, all the agents jointly optimize a global composite cost function which is the linear sum of individual cost functions composed of both smooth and non-smooth components. To solve this problem, we propose an asynchronous penalized proximal gradient (Asyn-PPG) algorithm, a variant of classical proximal gradient method, by considering the asynchronous update instants of the agents and communication delays in the network. Specifically, we consider a slot-based asynchronous network (SAN), where the whole time domain is split into sequential time slots and each agent is permitted to make multiple updates during a slot by accessing the historical state information of others. Moreover, we consider a set of global linear constraints and impose some violation penalties on the updating algorithms. By the Asyn-PPG algorithm, we will show that a periodically convergence with rate O(1/K) (K is the index of time slots) can be guaranteed if the coefficient of the penalties for all agents is synchronized at the end of the time slots and the step-size of the Asyn-PPG algorithm is properly determined. The feasibility of the proposed algorithm is verified by solving a consensus based distributed LASSO problem and a social welfare optimization problem in the electricity market respectively.

One-bit compressive sensing is popular in signal processing and communications due to the advantage of its low storage costs and hardware complexity. However, it has been a challenging task all along since only the one-bit (the sign) information is available to recover the signal. In this paper, we appropriately formulate the one-bit compressed sensing by a double-sparsity constrained optimization problem. The first-order optimality conditions via the newly introduced ? -stationarity for this nonconvex and discontinuous problem are established, based on which, a gradient projection subspace pursuit (GPSP) approach with global convergence and fast convergence rate is proposed. Numerical experiments against other leading solvers illustrate the high efficiency of our proposed algorithm in terms of the computation time and the quality of the signal recovery as well.

Non-convex optimization problems can be approximately solved via relaxation or local algorithms. For many practical problems such as optimal power flow (OPF) problems, both approaches tend to succeed in the sense that relaxation is usually exact and local algorithms usually converge to a global optimum. In this paper, we study conditions which are sufficient or necessary for such non-convex problems to simultaneously have exact relaxation and no spurious local optima. Those conditions help us explain the widespread empirical experience that local algorithms for OPF problems often work extremely well.

Congestion caused in the electrical network due to renewable generation can be effectively managed by integrating electric and thermal infrastructures, the latter being represented by large scale District Heating (DH) networks, often fed by large combined heat and power (CHP) plants. The CHP plants could further improve the profit margin of district heating multi-utilities by selling electricity in the power market by adjusting the ratio between generated heat and power. The latter is possible only for certain CHP plants, which allow decoupling the two commodities generation, namely the ones provided by two independent variables (degrees-of-freedom) or by integrating them with thermal energy storage and Power-to-Heat (P2H) units. CHP units can, therefore, help in the congestion management of the electricity network. A detailed mixed-integer linear programming (MILP) optimization model is introduced for solving the network-constrained unit commitment of integrated electric and thermal infrastructures. The developed model contains a detailed characterization of the useful effects of CHP units, i.e., heat and power, as a function of one and two independent variables. A lossless DC flow approximation models the electricity transmission network. The district heating model includes the use of gas boilers, electric boilers, and thermal energy storage. The conducted studies on IEEE 24 bus system highlight the importance of a comprehensive analysis of multi-energy systems to harness the flexibility derived from the joint operation of electric and heat sectors and managing congestion in the electrical network.

Subgradient and Newton algorithms for nonsmooth optimization require generalized derivatives to satisfy subtle approximation properties: conservativity for the former and semismoothness for the latter. Though these two properties originate in entirely different contexts, we show that in the semi-algebraic setting they are equivalent. Both properties for a generalized derivative simply require it to coincide with the standard directional derivative on the tangent spaces of some partition of the domain into smooth manifolds. An appealing byproduct is a new short proof that semi-algebraic maps are semismooth relative to the Clarke Jacobian.

Price-based revenue management is an important problem in operations management with many practical applications. The problem considers a retailer who sells a product (or multiple products) over T consecutive time periods and is subject to constraints on the initial inventory levels. While the optimal pricing policy could be obtained via dynamic programming, such an approach is sometimes undesirable because of high computational costs. Approximate policies, such as the re-solving heuristics, are often applied as computationally tractable alternatives. In this paper, we show the following two results. First, we prove that a natural re-solving heuristic attains O(1) regret compared to the value of the optimal policy. This improves the O(lnT) regret upper bound established in the prior work of \cite{jasin2014reoptimization}. Second, we prove that there is an Ω(lnT) gap between the value of the optimal policy and that of the fluid model. This complements our upper bound result by showing that the fluid is not an adequate information-relaxed benchmark when analyzing price-based revenue management algorithms.

We address an optimal reachability problem for a planar manipulator in a constrained environment. After introducing the optmization problem in full generality, we practically embed the geometry of the workspace in the problem, by considering some classes of obstacles. To this end, we present an analytical approximation of the distance function from the ellipse. We then apply our method to particular models of hyper-redundant and soft manipulators, by also presenting some numerical experiments.