Christopher Kirkbride

Index Policies for the Admission Control and Routing of Impatient Customers to Heterogeneous Service Stations

We propose a general Markovian model for the optimal control of admissions and subsequent routing of customers for service provided by a collection of heterogeneous stations. Queue-length information is available to inform all decisions. Admitted customers will abandon the system if required to wait too long for service. The optimisation goal is the maximisation of reward rate earned from service completions, net of the penalties paid whenever admission is denied, and the costs incurred upon every customer loss through impatience. We show that the system is indexable under mild conditions on model parameters and give an explicit construction of an index policy for admission control and routing founded on a proposal of Whittle for restless bandits. We are able to gain insights regarding the strength of performance of the index policy from the nature of solutions to the Lagrangian relaxation used to develop the indices. These insights are strengthened by the development of performance bounds. Although we are able to assert the optimality of the index heuristic in a range of asymptotic regimes, the performance bounds are also able to identify instances where its performance is relatively weak. Numerical studies are used to illustrate and support the theoretical analyses.

General notions of indexability for queueing control and asset management

We develop appropriately generalized notions of indexability for problems of dynamic resource allocation where the resource concerned may be assigned more flexibility than is allowed, for example, in classical multi-armed bandits. Most especially we have in mind the allocation of a divisible resource (manpower, money, equipment) to a collection of objects (projects) requiring it in cases where its over-concentration would usually be far from optimal. The resulting project indices are functions of both a resource level and a state. They have a simple interpretation as a fair charge for increasing the resource available to the project from the specified resource level when in the specified state. We illustrate ideas by reference to two model classes which are of independent interest. In the first, a pool of servers is assigned dynamically to a collection of service teams, each of which mans a service station. We demonstrate indexability under a natural assumption that the service rate delivered is increasing and concave in the team size. The second model class is a generalization of the spinning plates model for the optimal deployment of a divisible investment resource to a collection of reward generating assets. Asset indexability is established under appropriately drawn laws of diminishing returns for resource deployment. For both model classes numerical studies provide evidence that the proposed greedy index heuristic performs strongly.

Spinning plates and squad systems : Policies for bi-directional restless bandits

This paper concerns two families of Markov decision problem that fall within the family of (bi-directional) restless bandits, an intractable class of decision processes introduced by Whittle. The spinning plates problem concerns the optimal management of a portfolio of reward-generating assets whose yields grow with investment but otherwise tend to decline. In the model of asset exploitation called the squad system, the yield from an asset tends to decline when it is used but will recover when the asset is at rest. In all cases, simply stated conditions are given that guarantee indexability of the problem, together with conditions necessary and sufficient for its strict indexability. The index heuristics for asset activation that emerge from the analysis are assessed numerically and found to perform very strongly.

Stochastic scheduling: A short history of index policies and new approaches to index generation for dynamic resource allocation

In the 1970’s John Gittins discovered that multi-armed bandits, an important class of models for the dynamic allocation of a single key resource among a set of competing projects, have optimal solutions of index form. At each decision epoch such policies allocate the resource to whichever project has the largest Gittins index. Since the 1970’s, Gittins’ index result together with a range of developments and reformulations of it have constituted an influential stream of ideas and results contributing to research into the scheduling of stochastic objects. We give a brief account of many of the most important contributions to this work and proceed to describe how index theory has recently been developed to produce strongly performing heuristic policies for the dynamic allocation of a divisible resource to a collection of stochastic projects (or bandits). A limitation on this work concerns the need for the structural requirement of indexability which is notoriously difficult to establish. We introduce a general framework for the development of index policies for dynamic resource allocation which circumvents this difficulty. We utilise this framework to generate index policies for two model classes of independent interest. Their performance is evaluated in an extensive numerical study.

Index Policies for Shooting Problems

We consider a scenario in which a single Red wishes to shoot at a collection of Blue targets, one at a time, to maximise some measure of return obtained from Blues killed before Reds own (possible) demise. Such a situation arises in various military contexts, such as the conduct of air defence by Red in the face of Blue SEAD (suppression of enemy air defences). A class of decision processes called multiarmed bandits has been previously deployed to develop optimal policies for Red, in which she attaches a calibrating (Gittins) index to each Blue target and optimally shoots next at the Blue with the largest index value. The current paper seeks to elucidate how a range of developments of index theory are able to accommodate features of such problems, which are of practical military import. Such features include levels of risk to Red that are policy dependent, Red having imperfect information about the Blues she faces, an evolving population of Blue targets, and the possibility of Red disengagement. The paper concludes with a numerical study that both compares the performance of (optimal) index policies to a range of competitors and also demonstrates the value to Red of (optimal) disengagement.

Dynamic routing to heterogeneous collections of unreliable servers

We argue the importance of problems concerning the dynamic routing of tasks for service in environments where the servers have diverse characteristics and are subject to breakdown. We propose a general model in which both service times and repair times at each machine are i.i.d.with some general distribution. Routing decisions take account of queue lengths, machine states (up or down), the elapsed processing times of jobs in service and the times to date of any machine repairs in progress. We develop an approach to machine calibration which yields a machine index which is a function of all of the preceding information. The heuristic which routes all tasks to the machine of current smallest index performs outstandingly well. The approach of the paper is flexible and is capable of yielding strongly performing routing policies for a range of variants of the basic model. These include cases where job processing is lost at each breakdown and where the machine state may be only partially observed.

Approximate dynamic programming algorithms for multidimensional flexible production-inventory problems

An important issue in the manufacturing and supply chain literature concerns the optimisation of inventory decisions. Single-product inventory problems are widely studied and have been optimally solved under a variety of assumptions and settings. However, as systems become more complex, inventory decisions become more complicated for which the methods/approaches for optimising single inventory systems are incapable of deriving optimal policies. Manufacturing process flexibility provides an example of such a complex application area. Decisions involving the interrelated product inventories and production facilities form a highly multidimensional, non-decomposable system for which optimal policies cannot be readily obtained. We propose the methodology of approximate dynamic programming (ADP) to overcome the computational challenge imposed by this multidimensionality. Incorporating a sample backup simulation approach, ADP develops policies by utilising only a fraction of the computations required by classical dynamic programming. However, there are few studies in the literature that optimise production decisions in a stochastic, multi-factory, multi-product inventory system of this complexity. This paper aims to explore the feasibility and relevancy of ADP algorithms for this application. We present the results from numerical experiments that establish the strong performance of policies developed via temporal difference ADP algorithms in comparison to optimal policies and to policies derived from a deterministic approximation of the problem.

Approximate Dynamic Programming Algorithms for Multidimensional Inventory Optimization Problems

An important issue in the supply chain literature concerns the optimization of inventory decisions. Single-product inventory problems are widely studied and have been optimally solved under a variety of assumptions. However, as supply chain systems become more complex, inventory decisions become more complicated for which the methods/approaches for optimizing single-product inventory systems are incapable of deriving optimal policies. Manufacturing process flexibility provides an example of such complex application areas. Interrelated products and production facilities form a highly multidimensional, non-decomposable system for which optimal policies cannot be obtained by classical methods. We propose the methodology of Approximate Dynamic Programming (ADP) to overcome the computational challenge imposed by this multidimensionality. Incorporating a sample backup approach, ADP develops policies by utilizing only a fraction of the computations required by classical Dynamic Programming. However, there are no studies in the literature that optimize production decisions in a stochastic, multifactory, multiproduct inventory system of this complexity. This paper aims to explore the feasibility of ADP algorithms for this application. We present the results from a series of numerical experiments that establish the strong performance of policies developed via temporal difference ADP algorithms in comparison to optimal policies.

Allocation Models and Heuristics for the Outsourcing of Repairs for a Dynamic Warranty Population

We consider a scenario in which a large equipment manufacturer wishes to outsource the work involved in repairing purchased goods while under warranty. Several external service vendors are available for this work. We develop models and analyses to support decisions concerning how responsibility for the warranty population should be divided between them. These also allow the manufacturer to resolve related questions concerning, for example, whether the service capacities of the contracted vendors are sufficient to deliver an effective post-sales service. Static allocation models yield information concerning the proportions of the warranty population for which the vendors should be responsible overall. Dynamic allocation models enable consideration of how such overall workloads might be delivered to the vendors over time in a way which avoids excessive variability in the repair burden. We apply dynamic programming policy improvement to develop an effective dynamic allocation heuristic. This is evaluated numerically and is also used as a yardstick to assess two simple allocation heuristics suggested by static models. A dynamic greedy allocation heuristic is found to perform well. Dividing the workload equally among vendors with different service capacities can lead to serious losses.

Monotone policies and indexability for bidirectional restless bandits

Motivated by a wide range of applications, we consider a development of Whittles restless bandit model in which project activation requires a state-dependent amount of a key resource, which is assumed to be available at a constant rate. As many projects may be activated at each decision epoch as resource availability allows. We seek a policy for project activation within resource constraints which minimises an aggregate cost rate for the system. Project indices derived from a Lagrangian relaxation of the original problem exist provided the structural requirement of indexability is met. Verification of this property and derivation of the related indices is greatly simplified when the solution of the Lagrangian relaxation has a state monotone structure for each constituent project. We demonstrate that this is indeed the case for a wide range of bidirectional projects in which the project state tends to move in a different direction when it is activated from that in which it moves when passive. This is natural in many application domains in which activation of a project ameliorates its condition, which otherwise tends to deteriorate or deplete. In some cases the state monotonicity required is related to the structure of state transitions, while in others it is also related to the nature of costs. Two numerical studies demonstrate the value of the ideas for the construction of policies for dynamic resource allocation, most especially in contexts which involve a large number of projects.