Arthur F. Veinott

The Multi-Armed Bandit Problem: Decomposition and Computation

This paper is dedicated to our friend and mentor, Cyrus Derman, on the occasion of his 60th birthday. The multi-armed bandit problem arises in sequentially allocating effort to one of N projects and sequentially assigning patients to one of N treatments in clinical trials. Gittins and Jones Gittins, J. C., Jones, D. M. 1974. A dynamic allocation index for the sequential design of experiments. J. Gani, K. Sarkadi, L. Vince, eds. Progress in Statistics. European Meeting of Statisticians, 1972, 1, North Holland, Amsterdam, 241--266. have shown that one optimal policy for the N-project problem, an N-dimensional discounted Markov decision chain, is determined by the following largest-index rule. There is an index for each state of each given project that depends only on the data of that project. In each period one allocates effort to a project with largest current index. The purpose of this paper is to give a short proof of this result and a new characterization of the index of a project in state i, viz., as the maximum expected present value in state i for the restart-in-i problem in which, in each state and period, one either continues allocating effort to the project or immediately restarts the project in state i. Moreover, it is shown that an approximate largest-index rule yields an approximately optimal policy. These results lead to more efficient methods of computing the indices on-line and/or for sparse transition matrices in large state spaces than have been suggested heretofore. By using a suitable implementation of successive approximations, a policy whose expected present value is within 100e% of the maximum possible range of values of the indices can be found on-line with at most N + T-1TM operations where M is the number of operations required to calculate one approximation, T is the least integer majorizing the ratio ln e/ln a and 0

Send-and-Split Method for Minimum-Concave-Cost Network Flows

Many problems from inventory, production and capacity planning, and from network design, exhibit scale economies and can be formulated in terms of finding minimum-additive-concave-cost nonnegative network flows. We reduce the problem to an equivalent one in which the arc flow costs are nonnegative and give a dynamic-programming method, called the send-and-split method, to solve it. The main work of the method entails repeatedly solving set-splitting and minimum-cost-chain problems. In uncapacitated networks with n nodes, a arcs, and d + 1 demand nodes, i.e., nodes with nonzero exogenous demand, the algorithm requires up to n2-13d + s2d operations additions and comparisons where s = n log2n + 3a is the number of operations required to solve a minimum-cost-chain problem with nonnegative arc costs on the augmented graph formed by appending a node and an arc thereto from each node in the graph. If also the network is k-planar, i.e., the graph is planar with all demand nodes lying on the boundary of k faces, the method requires at most n2-kd3k + sd2k operations. The algorithm can be applied to capacitated networks because they can be reduced to equivalent uncapacitated ones. These results unify, significantly generalize e.g., to cyclic problems, and sometimes improve upon e.g., for tandem facilities known polynomial-time dynamic-programming algorithms for Wagner and Whitins Wagner, H. M., Whitin, T. M. 1958. Dynamic version of the economic lot size model. Management Sci.5 89--96. dynamic economic-order-quantity problem, Zangwills Zangwill, W. I. 1969. A backlogging model and a multi-echelon model of a dynamic economic lot size production system---A network approach. Management Sci.15 506--527. generalization to tandem facilities, and Veinotts Veinott, Jr., A. F. 1969. Minimum concave-cost solution of Leontief substitution models of multi-facility inventory systems. Oper. Res.17 262--291. increasing-capacity warehousing problem. The networks for the finite-period versions of these problems are each 1-planar. The method improves upon Zangwills Zangwill, W. I. 1968. Minimum concave cost flows in certain networks. Management Sci.14 429--450. related Oand running-time dynamic-programming method for finding minimum-additive-concave-cost non-negative flows in circuitless single-source networks. We also implement the method to solve in polynomial time the d + 1-demand-node and k-planar versions of the minimum-cost forest and Steiner problems in graphs. The running time for the d + 1-demand-node Steiner problem in graphs is comparable to that of Dreyfus and Wagners Dreyfus, S. E., Wagner, R. A. 1971. The Steiner problem in graphs. Networks1 195--207. method.

The Supporting Hyperplane Method for Unimodal Programming

This paper proposes an algorithm for maximizing a linear unimodal function over a compact convex set X in n-dimensional Euclidian space. The algorithm is a variant of the cutting-plane method of Cheney and Goldstein and of Kelley. In their method one maximizes the original linear unimodal function over successively refined approximations to X. Each approximation is the intersection of finitely many closed half spaces containing X. Any limit point of the sequence of solutions to the approximating problems solves the original problem. The principal new feature of our method is that each successive half space is defined by a hyperplane that supports X at a boundary point Zoutendijks MFD method also has this property. The particular boundary point is chosen as the point where the line drawn from a fixed interior point of X to the solution exterior to X to the most recent approximating problem pierces the boundary of X. Our method is applicable where X is described as the set on which finitely many unimodal functions are nonnegative. The cutting-plane method is applicable only under the more restrictive assumption that the above functions are concave. Our method is formulated so as to be applicable also to unimodal integer programs.

Polyhedral sets having a least element

For a fixedm × n matrixA, we consider the family of polyhedral setsXb ={x|Ax ≥ b}, b ∈ Rm, and prove a theorem characterizing, in terms ofA, the circumstances under which every nonemptyXb has a least element. In the special case whereA contains all the rows of ann × n identity matrix, the conditions are equivalent toAT being Leontief. Among the corollaries of our theorem, we show the linear complementarity problem always has a unique solution which is at the same time a least element of the corresponding polyhedron if and only if its matrix is square, Leontief, and has positive diagonals.

Integral Extreme Points

Abstract : It is shown that if A is an integral matrix having linearly independent rows, then the extreme points of the set of nonnegative solutions to Ax = b are integral for all integral b if and only if the determinant of every basis matrix is plus or minus 1. This provides a short proof of the Hoffman- Kruskal theorem characterizing unimodular matrices, i.e., matrices in which the determinant of each nonsingular submatrix is plus or minus 1. Their theorem is that if A is integral, then A is unimodular if and only if the extreme points of the set of nonnegative solutions to Ax = or b are integral for all integral b.

Substitutes, Complements and Ripples in Network Flows

We study the qualitative variation of minimum-cost network-flows and their associated costs with arc parameters. Demands are given at each node, flow is conserved, each arcs parameter lies in a lattice, and the flow cost is real or infinity-valued. Our main results are roughly as follows. If the flow cost is arc-additive, the problem can be decomposed into independent problems on each biconnected component of the graph. If also the flow cost is convex in the flow, the magnitude of the change in the optimal flow in arc a resulting from changing arc b s parameter diminishes the less biconnected a is to b . Arc a is less-biconnected-to b than is arc d if every simple cycle containing a and b also contains d . This relation is a quasi order with two distinct arcs being equivalent if and only if deleting them disconnects the graph. Hie Hasse diagram of the partial order of the induced equivalence classes is a tree with all arcs directed towards the class containing b . Two arcs are complements (resp., substitutes ) if every simple cycle containing both orients them in the same (resp., opposite) way. For example, two arcs are conformal , i.e., complements or substitutes, if they are either incident, or lie on a common face of a planar graph, or (as Dirac [Dirac, G. A. 1952. A property of 4-chromatic graphs and some remarks on critical graphs. J. London Math. Soc. 27 85--92.] and Duffin [Duffin, R. J. 1965. Topology of series-parallel networks. J. Math. Anal. Appl. 10 303--318.] have shown) lie in a series-parallel graph. If also each arc cost is (lattice) subadditive, then the optimal flow in arc a is nondecreasing (resp., nonincreasing) in the parameter of each complement (resp., substitute) of a . If moreover each parameter lies in a chain, then the minimum cost is superadditive in the parameters of a set of substitutes. Suppose in addition that the arc parameters are real and the arc costs are doubly subadditive. Then the absolute difference of the optimal flows in an arc corresponding to two monotonically-step-connected parameter vectors does not exceed the sum of the absolute differences of their elements. If further the arc costs are affine between successive integers and the difference between the two parameter vectors is a unit vector, then the difference between corresponding integer optimal flows is a unit simple circulation. This fact leads to a parametric algorithm for finding optimal flows. Finally, suppose instead that the flow cost is a sum of a subadditive flow cost for arcs in a set S of complements and a flow cost for arcs not in S that is arc-additive and convex in the flows therein. Then the optimal flow in each arc in S is nondecreasing in the parameters of those arcs, and the minimum cost is subadditive therein. Moreover, the optimal flow in each arc a not in S is nondecreasing (resp., nonincreasing) in the parameters of arcs in S if a is a complement (resp., substitute) of every arc in S .

Subextremal functions and lattice programming

Let M and N be the set of minimizers of a function f over respective subsets K and L of a lattice, with K being lower than L. This paper characterizes the class of functions f for which M is lower (resp., weakly lower, meet lower, join lower, chain lower) than N for all K lower than L. The resulting five classes of functions, called subextremal variants, have alternate characterizations by variants of the downcrossing-differences property, i.e., their first differences change sign at most once from plus to minus along complementary chains.

Discovering Hidden Totally Leontief Substitution Systems

A constructive procedure is given for determining the existence of and evaluating when it does exist a nonsingular matrix that transforms a system of linear equations in nonnegative variables into a totally Leontief substitution system. The computational effort involved is about that required to optimize the given m-row linear system with m + 1 different linear objective functions.

Staircase transportation problems with superadditive rewards and cumulative capacities

A cumulative-capacitated transportation problem is studied. The supply nodes and demand nodes are each chains. Shipments from a supply node to a demand node are possible only if the pair lies in a sublattice, or equivalently, in a staircase disjoint union of rectangles, of the product of the two chains. There are (lattice) superadditive upper bounds on the cumulative flows in all leading subrectangles of each rectangle. It is shown that there is a greatest cumulative flow formed by the natural generalization of the South-West Corner Rule that respects cumulative-flow capacities; it has maximum reward when the rewards are (lattice) superadditive; it is integer if the supplies, demands and capacities are integer; and it can be calculated myopically in linear time. The result is specialized to earlier work of Hoeffding (1940), Fréchet (1951), Lorentz (1953), Hoffman (1963) and Barnes and Hoffman (1985). Applications are given to extreme constrained bivariate distributions, optimal distribution with limited one-way product substitution and, generalizing results of Derman and Klein (1958), optimal sales with age-dependent rewards and capacities.

Substitutes, complements, and ripples in multicommodity flows on suspension graphs

We examine in this article when it is possible to predict, without numerical computation, the direction of change of optimal multicommodity flows on suspension graphs resulting from changes in arc-commodity parameters. Using results of Evans (Oper Res 26 (1978), 673–679) and of Soun and Truemper (SIAM J Algebr Discrete Meth 1 (1980), 348–358), the multicommodity flow problem on a graph that is two-isomorphic to a suspension graph is reduced to a single-commodity flow problem on an enlarged graph, called a “rolodex graph.” Such a reduction allows us to apply results of Granot and Veinott (Math Oper Res 10 (1985), 471–497), developed for single-commodity network-flow problems, to derive qualitative sensitivity analysis results for multicommodity flow problems on graphs which are two-isomorphic to suspension graphs.