Hugo P. Simão

An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application

We addressed the problem of developing a model to simulate at a high level of detail the movements of over 6,000 drivers for Schneider National, the largest truckload motor carrier in the United States. The goal of the model was not to obtain a better solution but rather to closely match a number of operational statistics. In addition to the need to capture a wide range of operational issues, the model had to match the performance of a highly skilled group of dispatchers while also returning the marginal value of drivers domiciled at different locations. These requirements dictated that it was not enough to optimize at each point in time (something that could be easily handled by a simulation model) but also over time. The project required bringing together years of research in approximate dynamic programming, merging math programming with machine learning, to solve dynamic programs with extremely high-dimensional state variables. The result was a model that closely calibrated against real-world operations and produced accurate estimates of the marginal value of 300 different types of drivers.

SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy

We address the problem of modeling energy resource allocation, including dispatch, storage, and the long-term investments in new technologies, capturing different sources of uncertainty such as energy from wind, demands, prices, and rainfall. We also wish to model long-term investment decisions in the presence of uncertainty. Accurately modeling the value of all investments, such as wind turbines and solar panels, requires handling fine-grained temporal variability and uncertainty in wind and solar in the presence of storage. We propose a modeling and algorithmic strategy based on the framework of approximate dynamic programming (ADP) that can model these problems at hourly time increments over an entire year or several decades. We demonstrate the methodology using both spatially aggregate and disaggregate representations of energy supply and demand. This paper describes the initial proof of concept experiments for an ADP-based model called SMART; we describe the modeling and algorithmic strategy and provide comparisons against a deterministic benchmark as well as initial experiments on stochastic data sets.

An Adaptive Dynamic Programming Algorithm for the Heterogeneous Resource Allocation Problem

We consider an aggregated version of a large-scale driver scheduling problem, derived from an application in less-than-truckload trucking, as a dynamic resource allocation problem. Drivers are aggregated into groups characterized by an attribute vector which capture the important attributes required to incorporate the work rules. The problem is very large: over 5,000 drivers and 30,000 loads in a four-day planning horizon. We formulate a problem that we call theheterogeneous resource allocation problem, which is more general than a classical multicommodity flow problem. Since the tasks have one-sided time windows, the problem is too large to even solve an LP relaxation. We formulate the problem as a multistage dynamic program and solve it using adaptive dynamic programming techniques. Since our problem is too large to solve using commercial solvers, we propose three independent benchmarks and demonstrate that our technique appears to be providing high-quality solutions in a reasonable amount of time.

Approximate dynamic programming for high dimensional resource allocation problems

There are wide arrays of discrete resource allocation problems (buffers in manufacturing, complex equipment in electric power, aircraft and locomotives in transportation) which need to be solved over time, under uncertainty. These can be formulated as dynamic programs, but typically exhibit high dimensional state, action and outcome variables (the three curses of dimensionality). For example, we have worked on problems where the dimensionality of these variables is in the ten thousand to one million range. We describe an approximation methodology for this problem class, and summarize the problem classes where the approach seems to be working well, and research challenges that we continue to face.

A Representational Paradigm for Dynamic Resource Transformation Problems

There are a host of complex operational problems arising in transportation and logistics which are characterized by dynamic information processes, complex operational characteristics and decentralized control structures. Yet, they are also optimization problems. The optimization community has made outstanding progress in the solution of large optimization problems when information processes are static (we do not model the arrival of new information) and when the entire problem can be viewed as being part of a single control structure. Not surprisingly, this technology has been extremely successful in applications such as planning airline operations which meet these requirements. This paper has grown out of the challenges we faced modeling complex operational problems arising in freight transportation and logistics, which are characterized by highly dynamic information processes, complex operational characteristics and decentralized control structures. Whereas people solve more traditional problems have struggled with the development of effective algorithms, we have struggled with the more basic challenge of simply modeling the problem. We feel that our ability to solve these problems is limited by the languages that we use to express them. Classical mathematical paradigms do not provide an easy and natural way to represent the optimization of these problems in the presence of dynamic information processes, or to capture the complexities of large scale operations. In particular, models do not capture the organization and flow of information in large organizations, preferring instead to assume the presence of a single, all-knowing decision-maker. As a result, most dynamic models posed in the literature are myopic or deterministic. The characteristics of more complex operations has spawned an extensive literature presenting models that are unique to a particular industry. For example, we solve airline fleet assignment problems (Hane et al. [22]), railroad car distribution problems (Jordan and Turnquist [26], Mendiratta and Turnquist [27], Haghani [21], and Herren [23], for example), the load matching problem of truckload trucking (Powell [33], Powell [34], Schrijver [40]), routing and scheduling problems in less-than-truckload trucking (Powell [32], Crainic and Roy [12]), the flow management problem in air traffic control (Andreatta and Romanin-Jacur [2] and Odoni [29]) and the management of ocean containers (Crainic, Gendreau and Dejax [11]). Even within an industry, rail car distribution is dif

Approximate dynamic programming for management of high‐value spare parts

Purpose – An aircraft manufacturer faces the problem of allocating inventory to a set of distributed warehouses in response to random, nonstationary demands. There is particular interest in managing high value, low volume spare parts which must be available to respond to low‐frequency demands in the form of random failures of major components. The aircraft fleet is young and in expansion. In addition, high‐value parts can be repaired, implying that they reenter the system after they are removed from an aircraft and refurbished. This paper aims to present a model and a solution approach to the problem of determining the inventory levels at each warehouse.Design/methodology/approach – The problem is solved using approximate dynamic programming (ADP), but this requires developing new methods for approximating value functions in the presence of low‐frequency observations.Findings – The model and solution approach have been implemented, tested and validated internally at the manufacturer through the analysis o...

Dynamic fleet management as a logistics queueing network

This paper introduces a new framework for modeling and solving dynamic fleet management problems, which we call the Logistics Queueing Network (LQN). A variety of problems in logistics involve the combined problem of moving freight from origin to destination while simultaneously managing the capacity required to move this freight. Standard formulations for real-world problems usually lead to intractably large linear programs. The LQN approach can take into account more real-world detail and is considerably faster than classical LP formulations. The solutions generated using the LQN approach are shown to be within a few percentage points of the LP optimal solutions depending on the size of the capacity fleets.

Approximate dynamic programming in transportation and logistics: a unified framework

Deterministic optimization has enjoyed a rich place in transportation and logistics, where it represents a mature field with established modeling and algorithmic strategies. By contrast, sequential stochastic optimization models (dynamic programs) have been plagued by the lack of a common modeling framework, and by algorithmic strategies that just do not seem to scale to real-world problems in transportation. This paper is designed as a tutorial of the modeling and algorithmic framework of approximate dynamic programming; however, our perspective on approximate dynamic programming is relatively new, and the approach is new to the transportation research community. We present a simple yet precise modeling framework that makes it possible to integrate most algorithmic strategies into four fundamental classes of policies, the design of which represents approximate solutions to these dynamic programs. The paper then uses problems in transportation and logistics to indicate settings in which each of the four classes of policies represents a natural solution strategy, highlighting the fact that the design of effective policies for these complex problems will remain an exciting area of research for many years. Along the way, we provide a link between dynamic programming, stochastic programming and stochastic search.

Numerical methods for simulating transient, stochastic queueing networks. I : Methodology

The study of stochastic networks of queues in transportation has been confined to the use of highly simplified analytical models or the development of large, computationally expensive Monte Carlo simulations. We introduce a discrete-time approach for simulating stochastic, transient networks of bulk queues that often arise in consolidation networks. This paper presents the state variables and equations required to model the problem, and introduces a set of approximations to produce a computationally tractable algorithm. A companion paper describes the results of extensive numerical experiments which test the accuracy of the approximations and the overall efficiency of the procedure.

Approximate Dynamic Programming Captures Fleet Operations for Schneider National

Schneider National needed a simulation model that would capture the dynamics of its fleet of over 6,000 long-haul drivers to determine where the company should hire new drivers, estimate the impact of changes in work rules, find the best way to manage Canadian drivers, and experiment with new ways to get drivers home. It needed a model that could perform as well as its experienced team of dispatchers and fleet managers. In developing our model, we had to simulate drivers and loads at a high level of detail, capturing both complex dynamics and multiple forms of uncertainty. We used approximate dynamic programming to produce realistic, high-quality decisions that capture the ability of dispatchers to anticipate the future impact of decisions. The resulting model closely calibrated against Schneiders historical performance, giving the company the confidence to base major policy decisions on studies performed using the model. These policy decisions helped Schneider to avoid costs of