Abraham P. George

Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming

We address the problem of determining optimal stepsizes for estimating parameters in the context of approximate dynamic programming. The sufficient conditions for convergence of the stepsize rules have been known for 50 years, but practical computational work tends to use formulas with parameters that have to be tuned for specific applications. The problem is that in most applications in dynamic programming, observations for estimating a value function typically come from a data series that can be initially highly transient. The degree of transience affects the choice of stepsize parameters that produce the fastest convergence. In addition, the degree of initial transience can vary widely among the value function parameters for the same dynamic program. This paper reviews the literature on deterministic and stochastic stepsize rules, and derives formulas for optimal stepsizes for minimizing estimation error. This formula assumes certain parameters are known, and an approximation is proposed for the case where the parameters are unknown. Experimental work shows that the approximation provides faster convergence than other popular formulas.

An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application

We addressed the problem of developing a model to simulate at a high level of detail the movements of over 6,000 drivers for Schneider National, the largest truckload motor carrier in the United States. The goal of the model was not to obtain a better solution but rather to closely match a number of operational statistics. In addition to the need to capture a wide range of operational issues, the model had to match the performance of a highly skilled group of dispatchers while also returning the marginal value of drivers domiciled at different locations. These requirements dictated that it was not enough to optimize at each point in time (something that could be easily handled by a simulation model) but also over time. The project required bringing together years of research in approximate dynamic programming, merging math programming with machine learning, to solve dynamic programs with extremely high-dimensional state variables. The result was a model that closely calibrated against real-world operations and produced accurate estimates of the marginal value of 300 different types of drivers.

SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy

We address the problem of modeling energy resource allocation, including dispatch, storage, and the long-term investments in new technologies, capturing different sources of uncertainty such as energy from wind, demands, prices, and rainfall. We also wish to model long-term investment decisions in the presence of uncertainty. Accurately modeling the value of all investments, such as wind turbines and solar panels, requires handling fine-grained temporal variability and uncertainty in wind and solar in the presence of storage. We propose a modeling and algorithmic strategy based on the framework of approximate dynamic programming (ADP) that can model these problems at hourly time increments over an entire year or several decades. We demonstrate the methodology using both spatially aggregate and disaggregate representations of energy supply and demand. This paper describes the initial proof of concept experiments for an ADP-based model called SMART; we describe the modeling and algorithmic strategy and provide comparisons against a deterministic benchmark as well as initial experiments on stochastic data sets.

Approximate dynamic programming for high dimensional resource allocation problems

There are wide arrays of discrete resource allocation problems (buffers in manufacturing, complex equipment in electric power, aircraft and locomotives in transportation) which need to be solved over time, under uncertainty. These can be formulated as dynamic programs, but typically exhibit high dimensional state, action and outcome variables (the three curses of dimensionality). For example, we have worked on problems where the dimensionality of these variables is in the ten thousand to one million range. We describe an approximation methodology for this problem class, and summarize the problem classes where the approach seems to be working well, and research challenges that we continue to face.

The Effect of Robust Decisions on the Cost of Uncertainty in Military Airlift Operations

There are a number of sources of randomness that arise in military airlift operations. However, the cost of uncertainty can be difficult to estimate, and is easy to overestimate if we use simplistic decision rules. Using data from Canadian military airlift operations, we study the effect of uncertainty in customer demands as well as aircraft failures, on the overall cost. The system is first analyzed using the types of myopic decision rules widely used in the research literature. The performance of the myopic policy is then compared to the results obtained using robust decisions that account for the uncertainty of future events. These are obtained by modeling the problem as a dynamic program, and solving Bellman’s equations using approximate dynamic programming. The experiments show that even approximate solutions to Bellman’s equations produce decisions that reduce the cost of uncertainty.

Approximate Dynamic Programming Captures Fleet Operations for Schneider National

Schneider National needed a simulation model that would capture the dynamics of its fleet of over 6,000 long-haul drivers to determine where the company should hire new drivers, estimate the impact of changes in work rules, find the best way to manage Canadian drivers, and experiment with new ways to get drivers home. It needed a model that could perform as well as its experienced team of dispatchers and fleet managers. In developing our model, we had to simulate drivers and loads at a high level of detail, capturing both complex dynamics and multiple forms of uncertainty. We used approximate dynamic programming to produce realistic, high-quality decisions that capture the ability of dispatchers to anticipate the future impact of decisions. The resulting model closely calibrated against Schneiders historical performance, giving the company the confidence to base major policy decisions on studies performed using the model. These policy decisions helped Schneider to avoid costs of

An adaptive-learning framework for semi-cooperative multi-agent coordination

30 million by identifying problems with a new driver-management policy, achieve annual savings of

Value Function Approximation using Multiple Aggregation for Multiattribute Resource Management

5 million by identifying the best driver domiciles, reduce the number of late deliveries by more than 50 percent by analyzing service commitment policies, and save

Value Function Approximation using Hierarchical Aggregation for Multiattribute Resource Management

3.8 million annually by reducing training expenses for new border-crossing regulations.

of Robust Decisions on the Cost of Uncertainty in Military Airlift Operations

Complex problems involving multiple agents exhibit varying degrees of cooperation. The levels of cooperation might reflect both differences in information as well as differences in goals. In this research, we develop a general mathematical model for distributed, semi-cooperative planning and suggest a solution strategy which involves decomposing the system into subproblems, each of which is specified at a certain period in time and controlled by an agent. The agents communicate marginal values of resources to each other, possibly with distortion. We design experiments to demonstrate the benefits of communication between the agents and show that, with communication, the solution quality approaches that of the ideal situation where the entire problem is controlled by a single agent.