Nahum Shimkin

Competitive routing in multiuser communication networks

The authors consider a communication network shared by several selfish users. Each user seeks to optimize its own performance by controlling the routing of its given flow demand, giving rise to a noncooperative game. They investigate the Nash equilibrium of such systems. For a two-node multiple links system, uniqueness of the Nash equilibrium is proven under reasonable convexity conditions. It is shown that this Nash equilibrium point possesses interesting monotonicity properties. For general networks, these convexity conditions are not sufficient for guaranteeing uniqueness, and a counterexample is presented. Nonetheless, uniqueness of the Nash equilibrium for general topologies is established under various assumptions. >

Competitive routing in networks with polynomial costs

We study a class of noncooperative general topology networks shared by N users. Each user has a given flow which it has to ship from a source to a destination. We consider a class of polynomial link cost functions adopted originally in the context of road traffic modeling, and show that these costs have appealing properties that lead to predictable and efficient network flows. In particular, we show that the Nash equilibrium is unique, and is moreover efficient. These properties make the polynomial cost structure attractive for traffic regulation and link pricing in telecommunication networks. We finally discuss the computation of the equilibrium in the special case of the affine cost structure for a topology of parallel links.

Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning

We present the Q-Cut algorithm, a graph theoretic approach for automatic detection of sub-goals in a dynamic environment, which is used for acceleration of the Q-Learning algorithm. The learning agent creates an on-line map of the process history, and uses an efficient Max-Flow/Min-Cut algorithm for identifying bottlenecks. The policies for reaching bottlenecks are separately learned and added to the model in a form of options (macro-actions). We then extend the basic Q-Cut algorithm to the Segmented Q-Cut algorithm, which uses previously identified bottlenecks for state space partitioning, necessary for finding additional bottlenecks in complex environments. Experiments showsign ificant performance improvements, particulary in the initial learning phase.

Basis Function Adaptation in Temporal Difference Reinforcement Learning

Reinforcement Learning (RL) is an approach for solving complex multi-stage decision problems that fall under the general framework of Markov Decision Problems (MDPs), with possibly unknown parameters. Function approximation is essential for problems with a large state space, as it facilitates compact representation and enables generalization. Linear approximation architectures (where the adjustable parameters are the weights of pre-fixed basis functions) have recently gained prominence due to efficient algorithms and convergence guarantees. Nonetheless, an appropriate choice of basis function is important for the success of the algorithm. In the present paper we examine methods for adapting the basis function during the learning process in the context of evaluating the value function under a fixed control policy. Using the Bellman approximation error as an optimization criterion, we optimize the weights of the basis function while simultaneously adapting the (non-linear) basis function parameters. We present two algorithms for this problem. The first uses a gradient-based approach and the second applies the Cross Entropy method. The performance of the proposed algorithms is evaluated and compared in simulations.

Adaptive Behavior of Impatient Customers in Tele-Queues: Theory and Empirical Support

We address the modeling and analysis of abandonments from a queue that is invisible to its occupants. Such queues arise in remote service systems, notably the Internet and telephone call centers; hence, we refer to them as tele-queues. A basic premise of this paper is that customers adapt their patience modeled by an abandonment-time distribution to their service expectations, in particular to their anticipated waiting time. We present empirical support for that hypothesis, and propose an M/M/m-based model that incorporates adaptive customer behavior. In our model, customer patience depends on the mean waiting time in the queue. We characterize the resulting system equilibrium namely, the operating point in steady state, and establish its existence and uniqueness when changes in customer patience are bounded by the corresponding changes in their anticipated waiting time. The feasibility of multiple system equilibria is illustrated when this condition is violated. Finally, a dynamic learning model is proposed where customer expectations regarding their waiting time are formed through accumulated experience. We demonstrate, via simulation, convergence to the theoretically anticipated equilibrium, while addressing certain issues related to censored-sampling that arise because of abandonments.

The Impact of Delay Announcements in Many-Server Queues with Abandonment

This is a supplement to the main paper, having the same title. In this work we develop methods to study the impact upon steady-state performance of delay announcements made to arriving customers in a many-server queue with customer abandonment. We assume that the queue is not visible to waiting customers, as in most customer contact centers, when contact is made by telephone, email or instant messaging. We propose simple robust announcement schemes: (i) the delay of the last served (DLS) customer and (ii) a flxed delay announcement (FDA) based on an appropriate long-run average delay. For any single-number delay announcement made immediately upon arrival, customers may balk or have new abandonment behavior as a function of the announced delay. We introduce a model of that customer response. To perform a rough-cut performance analysis, prior to detailed simulation, we introduce a ∞uid model, which provides an approximate and highly simplifled description for large systems in an overloaded regime. In the ∞uid model, all customers are faced with the same delay and consequently can be given the same delay announcement. That property motivates considering a second approximation scheme: an equilibrium flxed delay announcement in the stochastic model, which we compute approximately using an iterative numerical algorithm (INA). We show that these two approximate descriptions of aggregate performance are efiective by comparing to simulations. We simulate systems with state-dependent DLS announcements and we iterate over simulations using flxed delay announcements. Here we present additional material, supplementing the main paper.

Competitive routing in multi-user communication networks

A communication network shared by several selfish users is considered. Each user seeks to optimize its own performance by controlling the routing of its given flow demand, giving rise to a noncooperative game. The Nash equilibrium of such systems is investigated. For a two-node multiple-link system, the uniqueness of the Nash equilibrium is proved under reasonable convexity conditions. It is shown that this Nash equilibrium point possesses interesting monotonicity properties. For general networks, the uniqueness of the Nash equilibrium is established under various assumptions.<<ETX>>

Competitive routing in networks with polynomial cost

We study a class of noncooperative general topology networks shared by N users. Each user has a given flow which it has to ship from a source to a destination. We consider a class of polynomial link cost functions, adopted originally in the context of road traffic modeling, and show that these costs have appealing properties that lead to predictable and efficient network flows. In particular, we show that the Nash equilibrium is unique, and is moreover efficient, i.e., it coincides with the solution of a corresponding global optimization problem with a single user. These properties make the cost structure attractive for traffic regulation and link pricing in telecommunication networks. We finally discuss the computation of the equilibrium in the special case of the affine cost structure for a topology of parallel links.

Routing into Two Parallel Links

We study a class of noncooperative networks where N users send traffic to a destination node over two links with given capacities in such a way that a Nash equilibrium is achieved. Under a linear cost structure for the individual users, we obtain several dynamic policy adjustment schemes for the online computation of the Nash equilibrium and study their local convergence properties. These policy adjustment schemes require minimum information on the part of each user regarding the cost?utility functions of the others.

Markov Decision Processes with Arbitrary Reward Processes

We consider a learning problem where the decision maker interacts with a standard Markov decision process, with the exception that the reward functions vary arbitrarily over time. We show that, against every possible realization of the reward process, the agent can perform as well---in hindsight---as every stationary policy. This generalizes the classical no-regret result for repeated games. Specifically, we present an efficient online algorithm---in the spirit of reinforcement learning---that ensures that the agents average performance loss vanishes over time, provided that the environment is oblivious to the agents actions. Moreover, it is possible to modify the basic algorithm to cope with instances where reward observations are limited to the agents trajectory. We present further modifications that reduce the computational cost by using function approximation and that track the optimal policy through infrequent changes.