Cristiano Cervellera

Optimization of a large-scale water reservoir network by stochastic dynamic programming with efficient state space discretization

A numerical solution to a 30-dimensional water reservoir network optimization problem, based on stochastic dynamic programming, is presented. In such problems the amount of water to be released from each reservoir is chosen to minimize a nonlinear cost (or maximize benefit) function while satisfying proper constraints. Experimental results show how dimensionality issues, given by the large number of basins and realistic modeling of the stochastic inflows, can be mitigated by employing neural approximators for the value functions, and efficient discretizations of the state space, such as orthogonal arrays, Latin hypercube designs and low-discrepancy sequences.

Modeling and Feedback Control for Resource Allocation and Performance Analysis in Container Terminals

A dynamic discrete-time model of container flows in maritime terminals is proposed as a system of queues. Such queues are controlled via input variables that account for the use of the available resources given by the capacities of the handling machines used to move containers inside a terminal. Two feedback control strategies for the allocation of such resources are described. The first consists of a resource assignment that is proportional to the corresponding queue lengths; in the second, the assignment is obtained by the one-step-ahead optimization of a performance cost function according to a myopic approach. Simulation results are reported to compare such methodologies for the purpose of sensitivity and scenario analyses in the management of a maritime terminal.

Neural network and regression spline value function approximations for stochastic dynamic programming

Dynamic programming is a multi-stage optimization method that is applicable to many problems in engineering. A statistical perspective of value function approximation in high-dimensional, continuous-state stochastic dynamic programming (SDP) was first presented using orthogonal array (OA) experimental designs and multivariate adaptive regression splines (MARS). Given the popularity of artificial neural networks (ANNs) for high-dimensional modeling in engineering, this paper presents an implementation of ANNs as an alternative to MARS. Comparisons consider the differences in methodological objectives, computational complexity, model accuracy, and numerical SDP solutions. Two applications are presented: a nine-dimensional inventory forecasting problem and an eight-dimensional water reservoir problem. Both OAs and OA-based Latin hypercube experimental designs are explored, and OA space-filling quality is considered.

Deterministic design for neural network learning: an approach based on discrepancy

The general problem of reconstructing an unknown function from a finite collection of samples is considered, in case the position of each input vector in the training set is not fixed beforehand but is part of the learning process. In particular, the consistency of the empirical risk minimization (ERM) principle is analyzed, when the points in the input space are generated by employing a purely deterministic algorithm (deterministic learning). When the output generation is not subject to noise, classical number-theoretic results, involving discrepancy and variation, enable the establishment of a sufficient condition for the consistency of the ERM principle. In addition, the adoption of low-discrepancy sequences enables the achievement of a learning rate of O(1/L), with L being the size of the training set. An extension to the noisy case is provided, which shows that the good properties of deterministic learning are preserved, if the level of noise at the output is not high. Simulation results confirm the validity of the proposed approach.

Design of Asymptotic Estimators: An Approach Based on Neural Networks and Nonlinear Programming

A methodology to design state estimators for a class of nonlinear continuous-time dynamic systems that is based on neural networks and nonlinear programming is proposed. The estimator has the structure of a Luenberger observer with a linear gain and a parameterized (in general, nonlinear) function, whose argument is an innovation term representing the difference between the current measurement and its prediction. The problem of the estimator design consists in finding the values of the gain and of the parameters that guarantee the asymptotic stability of the estimation error. Toward this end, if a neural network is used to take on this function, the parameters (i.e., the neural weights) are chosen, together with the gain, by constraining the derivative of a quadratic Lyapunov function for the estimation error to be negative definite on a given compact set. It is proved that it is sufficient to impose the negative definiteness of such a derivative only on a suitably dense grid of sampling points. The gain is determined by solving a Lyapunov equation. The neural weights are searched for via nonlinear programming by minimizing a cost penalizing grid-point constraints that are not satisfied. Techniques based on low-discrepancy sequences are applied to deal with a small number of sampling points, and, hence, to reduce the computational burden required to optimize the parameters. Numerical results are reported and comparisons with those obtained by the extended Kalman filter are made

Efficient sampling in approximate dynamic programming algorithms

Abstract Dynamic Programming (DP) is known to be a standard optimization tool for solving Stochastic Optimal Control (SOC) problems, either over a finite or an infinite horizon of stages. Under very general assumptions, commonly employed numerical algorithms are based on approximations of the cost-to-go functions, by means of suitable parametric models built from a set of sampling points in the d-dimensional state space. Here the problem of sample complexity, i.e., how “fast” the number of points must grow with the input dimension in order to have an accurate estimate of the cost-to-go functions in typical DP approaches such as value iteration and policy iteration, is discussed. It is shown that a choice of the sampling based on low-discrepancy sequences, commonly used for efficient numerical integration, permits to achieve, under suitable hypotheses, an almost linear sample complexity, thus contributing to mitigate the curse of dimensionality of the approximate DP procedure.

A comparison of global and semi-local approximation in T-stage stochastic optimization

The paper presents a comparison between two different flavors of nonlinear models to be used for the approximate solution of T-stage stochastic optimization (TSO) problems, a typical paradigm of Markovian decision processes. Specifically, the well-known class of neural networks is compared with a semi-local approach based on kernel functions, characterized by less demanding computational requirements. To this purpose, two alternative methods for the numerical solution of TSO are considered, one corresponding to the classic approximate dynamic programming (ADP) and the other based on a direct optimization of the optimal control functions, introduced here for the first time. Advantages and drawbacks in the TSO context of the two classes of approximators are analyzed, in terms of computational burden and approximation capabilities. Then, their performances are evaluated through simulations in two important high-dimensional TSO test cases, namely inventory forecasting and water reservoirs management.

Efficient kernel models for learning and approximate minimization problems

This paper investigates techniques for reducing the computational burden of local learning methods relying on kernel functions in the framework of approximate minimization, i.e., when they are employed to find the minimum of a given cost functional. The considered approach is based on an optimal choice of the kernel width parameters through the minimization of an empirical cost and can provide a solution to important problems, such as function approximation and multistage optimization. However, when the stored data are too many, the kernel model output evaluation can take a long time, making local learning unsuited to contexts where a fast function evaluation is required. At the same time, the training procedure to obtain the kernel widths can become too demanding as well. Here it is shown that a large saving in the computational effort can be achieved by considering subsets of the available data suitably chosen according to different criteria. An analysis of the performance of the new approach is provided. Then, simulation results show in practice the effectiveness of the proposed techniques when applied to learning and approximate minimization problems.

Low-discrepancy sampling for approximate dynamic programming with local approximators

Approximate dynamic programming (ADP) relies, in the continuous-state case, on both a flexible class of models for the approximation of the value functions and a smart sampling of the state space for the numerical solution of the recursive Bellman equations. In this paper, low-discrepancy sequences, commonly employed for number-theoretic methods, are investigated as a sampling scheme in the ADP context when local models, such as the Nadaraya-Watson (NW) ones, are employed for the approximation of the value function. The analysis is carried out both from a theoretical and a practical point of view. In particular, it is shown that the combined use of low-discrepancy sequences and NW models enables the convergence of the ADP procedure. Then, the regular structure of the low-discrepancy sampling is exploited to derive a method for automatic selection of the bandwidth of NW models, which yields a significant saving in the computational effort with respect to the standard cross validation approach. Simulation results concerning an inventory management problem are presented to show the effectiveness of the proposed techniques.

Predictive Control of Container Flows in Maritime Intermodal Terminals

Predictive control is investigated as a paradigm for the allocation of handling resources to transfer containers inside intermodal terminals. The decisions on the allocation of such resources are derived from the minimization of performance cost functions that measure the lay times of carriers over a forward horizon basing on a model of the container flows. Such a model allows one to take advantage of the information available in real time on the arrival or departure of carriers with the corresponding amounts of containers scheduled for loading or unloading. The resulting strategy of resource allocation can be regarded as a feedback control law and is obtained by solving nonlinear programming problems online. Since the computation may be too expensive, a technique based on the idea of approximating offline such a law is proposed. The approximation is performed by using neural networks, which allow one to construct an approximate feedback controller and generate the corresponding online control actions with a negligible computational burden. The effectiveness of the approach is shown via simulations in a case study.