Sumit Kunnumkal

Using Stochastic Approximation Methods to Compute Optimal Base-Stock Levels in Inventory Control Problems

In this paper, we consider numerous inventory control problems for which the base-stock policies are known to be optimal, and we propose stochastic approximation methods to compute the optimal base-stock levels. The existing stochastic approximation methods in the literature guarantee that their iterates converge, but not necessarily to the optimal base-stock levels. In contrast, we prove that the iterates of our methods converge to the optimal base-stock levels. Moreover, our methods continue to enjoy the well-known advantages of the existing stochastic approximation methods. In particular, they only require the ability to obtain samples of the demand random variables, rather than to compute expectations explicitly, and they are applicable even when the demand information is censored by the amount of available inventory.

Computing Time-Dependent Bid Prices in Network Revenue Management Problems

We propose a new method to compute bid prices in network revenue management problems. The novel aspect of our method is that it naturally provides dynamic bid prices that depend on how much time is left until departure. We show that our method provides an upper bound on the optimal total expected revenue and that this upper bound is tighter than the one provided by the widely known deterministic linear programming approach. Furthermore, it is possible to use the bid prices computed by our method as a starting point in a dynamic programming decomposition-like idea to decompose the network revenue management problem by the flight legs and to obtain dynamic and capacity-dependent bid prices. Our computational experiments indicate that the proposed method improves on many standard benchmarks.

A stochastic approximation method for the single-leg revenue management problem with discrete demand distributions

We consider the problem of optimally allocating the seats on a single flight leg to the demands from multiple fare classes that arrive sequentially. It is well-known that the optimal policy for this problem is characterized by a set of protection levels. In this paper, we develop a new stochastic approximation method to compute the optimal protection levels under the assumption that the demand distributions are not known and we only have access to the samples from the demand distributions. The novel aspect of our method is that it works with the nonsmooth version of the problem where the capacity can only be allocated in integer quantities. We show that the sequence of protection levels generated by our method converges to a set of optimal protection levels with probability one. We discuss applications to the case where the demand information is censored by the seat availability. Computational experiments indicate that our method is especially advantageous when the total expected demand exceeds the capacity by a significant margin and we do not have good a priori estimates of the optimal protection levels.

A Randomized Linear Programming Method for Network Revenue Management with Product-Specific No-Shows

Revenue management practices often include overbooking capacity to account for customers who make reservations but do not show up. In this paper, we consider the network revenue management problem with no-shows and overbooking, where the show-up probabilities are specific to each product. No-show rates differ significantly by product (for instance, each itinerary and fare combination for an airline) as sale restrictions and the demand characteristics vary by product. However, models that consider no-show rates by each individual product are difficult to handle because the state-space in dynamic programming formulations (or the variable space in approximations) increases significantly. In this paper, we propose a randomized linear program to jointly make the capacity control and overbooking decisions with product-specific no-shows. We establish that our formulation gives an upper bound on the optimal expected total profit, and our upper bound is tighter than a deterministic linear programming upper bound that appears in the existing literature. Furthermore, we show that our upper bound is asymptotically tight in a regime where the leg capacities and the expected demand is scaled linearly with the same rate. We also describe how the randomized linear program can be used to obtain a bid price control policy. Computational experiments indicate that our approach is quite fast, is able to scale to industrial problems, and can provide significant improvements over standard benchmarks.

Linear programming based decomposition methods for inventory distribution systems

We consider an inventory distribution system consisting of one warehouse and multiple retailers. The retailers face random demand and are supplied by the warehouse. The warehouse replenishes its stock from an external supplier. The objective is to minimize the total expected replenishment, holding and backlogging cost over a finite planning horizon. The problem can be formulated as a dynamic program, but this dynamic program is difficult to solve due to its high dimensional state variable. It has been observed in the earlier literature that if the warehouse is allowed to ship negative quantities to the retailers, then the problem decomposes by the locations. One way to exploit this observation is to relax the constraints that ensure the nonnegativity of the shipments to the retailers by associating Lagrange multipliers with them, which naturally raises the question of how to choose a good set of Lagrange multipliers. In this paper, we propose efficient methods that choose a good set of Lagrange multipliers by solving linear programming approximations to the inventory distribution problem. Computational experiments indicate that the inventory replenishment policies obtained by our approach can outperform several standard benchmarks by significant margins.

On a Piecewise-Linear Approximation for Network Revenue Management

The network revenue management (RM) problem arises in airline, hotel, media, and other industries where the sale products use multiple resources. It can be formulated as a stochastic dynamic program, but the dynamic program is computationally intractable because of an exponentially large state space, and a number of heuristics have been proposed to approximate its value function. In this paper we show that the piecewise-linear approximation to the network RM dynamic program is tractable; specifically we show that the separation problem of the approximation can be solved as a relatively compact linear program. Moreover, the resulting compact formulation of the approximate dynamic program turns out to be exactly equivalent to the Lagrangian relaxation of the dynamic program, an earlier heuristic method proposed for the same problem. We perform a numerical comparison of solving the problem by generating separating cuts or as our compact linear program. We discuss extensions to versions of the network RM problem with overbooking as well as the difficulties of extending it to the choice model of network revenue RM.

Exploiting the Structural Properties of the Underlying Markov Decision Problem in the Q-Learning Algorithm

This paper shows how to exploit the structural properties of the underlying Markov decision problem to improve the convergence behavior of the Q-learning algorithm. In particular, we consider infinite-horizon discounted-cost Markov decision problems where there is a natural ordering between the states of the system and the value function is known to be monotone in the state. We propose a new variant of the Q-learning algorithm that ensures that the value function approximations obtained during the intermediate iterations are also monotone in the state. We establish the convergence of the proposed algorithm and experimentally show that it significantly improves the convergence behavior of the standard version of the Q-learning algorithm.

On upper bounds for assortment optimization under the mixture of multinomial logit models

The assortment optimization problem under the mixture of multinomial logit models is NP-complete and there are different approximation methods to obtain upper bounds on the optimal expected revenue. In this paper, we analytically compare the upper bounds obtained by the different approximation methods. We propose a new, tractable approach to construct an upper bound on the optimal expected revenue and show that it obtains the tightest bound among the existing tractable approaches in the literature to obtain upper bounds.

A stochastic approximation method with max-norm projections and its applications to the Q-learning algorithm

In this article, we develop a stochastic approximation method to solve a monotone estimation problem and use this method to enhance the empirical performance of the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We begin by considering a monotone estimation problem where we want to estimate the expectation of a random vector, η. We assume that the components of E {η} are known to be in increasing order. The stochastic approximation method that we propose is designed to exploit this information by projecting its iterates onto the set of vectors with increasing components. The novel aspect of the method is that it uses projections with respect to the max norm. We show the almost sure convergence of the stochastic approximation method. After this result, we consider the Q-learning algorithm when applied to Markov decision problems with monotone value functions. We study a variant of the Q-learning algorithm that uses projections to ensure that the value function approximation obtained at each iteration is also monotone. Computational results indicate that the performance of the Q-learning algorithm can be improved significantly by exploiting the monotonicity property of the value functions.

Technical Note—A Note on Relaxations of the Choice Network Revenue Management Dynamic Program

In recent years, several approximation methods have been proposed for the choice network revenue management problem. These approximation methods are proposed because the dynamic programming formulation of the choice network revenue management problem is intractable even for moderately sized instances. In this paper, we consider three approximation methods that obtain upper bounds on the value function, namely, the choice deterministic linear program (CDLP), the affine approximation (AF), and the piecewise-linear approximation (PL). It is known that the piecewise-linear approximation bound is tighter than the affine bound, which in turn is tighter than CDLP. In this paper, we prove bounds on how much the affine and piecewise-linear approximations can tighten CDLP. We show (i) the gap between the AF and CDLP bounds is at most a factor of 1+1/(mini{ri1}), where ri1>0 are the resource capacities, and (ii) the gap between the piecewise-linear and CDLP bounds is within a factor of 2. Moreover, we show that thes...