Peter Marbach

Call admission control and routing in integrated services networks using neuro-dynamic programming

We consider the problem of call admission control (CAC) and routing in an integrated services network that handles several classes of calls of different value and with different resource requirements. The problem of maximizing the average value of admitted calls per unit time (or of revenue maximization) is naturally formulated as a dynamic programming problem, but is too complex to allow for an exact solution. We use methods of neuro-dynamic programming (NDP) [reinforcement learning (RL)], together with a decomposition approach, to construct dynamic (state-dependent) call admission control and routing policies. These policies are based on state-dependent link costs, and a simulation-based learning method is employed to tune the parameters that define these link costs. A broad set of experiments shows the robustness of our policy and compares its performance with a commonly used heuristic.

Bandwidth allocation in ad hoc networks: a price-based approach

Pricing is considered as a means to stimulate cooperation in ad hoc networks: users can charge other users a price for relaying their data packets. Assuming that users set prices to maximize their own net benefit, we propose an iterative price and rate adaption algorithm. We show that this algorithm converges to a socially optimal bandwidth allocation. We use a numerical case study to illustrate our results.

Cooperation in wireless ad hoc networks: a market-based approach

We consider a market-based approach to stimulate cooperation in ad hoc networks where nodes charge a price for relaying data packets. Assuming that nodes set prices to maximize their own net benefit, we characterize the equilibria of the resulting market. In addition, we propose an iterative algorithm for the nodes to adapt their price and rate allocation, and study its convergence behavior. We use a numerical case study to illustrate our results.

Priority service and max-min fairness

We study a priority service where users are free to choose the priority of their traffic, but are charged accordingly by the network. We assume that each user chooses priorities to maximize its own net benefit, and model the resulting interaction among users as a noncooperative game. We show that there exists an unique equilibrium for this game and that in equilibrium the bandwidth allocation is weighted max-min fair.

Analysis of a static pricing scheme for priority services

We analyze a static pricing scheme for priority services. Users are free to choose the priority of their traffic but are charged accordingly. Using a game theoretic framework, we study the case where users choose priorities to maximize their net benefit. For the single link case, we show that there always exists an equilibrium for the corresponding game; however, the equilibrium is not necessarily unique. Furthermore, we show that packet loss in equilibrium can be expressed as a function of the prices associated with the different priority classes. We provide a numerical case study to illustrate our results.

On Wireless Social Community Networks

Wireless social community networks are emerging as a new alternative to providing wireless data access in urban areas. By relying on users in the network deployment, a wireless community can rapidly deploy a high-quality data access infrastructure in an inexpensive way. But, the coverage of such a network is limited by the set of access points deployed by the users. Currently, it is not clear if this paradigm can serve as a replacement of existing centralized networks operating in licensed bands (such as cellular networks) or if it should be considered as a complimentary service only, with limited coverage. This question currently concerns many wireless network operators. In this paper, we study the dynamics of wireless social community networks by using a simple analytical model. In this model, users choose their service provider based on the subscription fee and the offered coverage. We show how the evolution of social community networks depends on their initial coverage, the subscription fee, and the user preferences for coverage. We conclude that by using an efficient static or dynamic pricing strategy, the wireless social community can obtain a high coverage. Using a game-theoretic approach, we then study a case where the mobile users can choose between the services provided by a licensed band operator and those of a social community. We show that for specific distribution of user preferences, there exists a Nash equilibrium for this non-cooperative game.

Throughput-optimal random access with order-optimal delay

In this paper, we consider CSMA policies for scheduling packet transmissions in multihop wireless networks with one-hop traffic. The main contribution of the paper is to propose a novel CSMA policy, called Unlocking CSMA (U-CSMA), that enables to obtain both high throughput and low packet delays in large wireless networks. More precisely, we show that for torus interference graph topologies with one-hop traffic, U-CSMA is throughput optimal and achieves order-optimal delay. For one-hop traffic, the delay performance is defined to be order-optimal if the delay stays bounded as the network-size increases. Simulations that we conducted suggest that (a) U-CSMA is throughput-optimal and achieves order-optimal delay for general geometric interference graphs and (b) that U-CSMA can be combined with congestion control algorithms to maximize the network-wide utility and obtain order-optimal delay. To the best of our knowledge, this is the first time that a simple distributed scheduling policy has been proposed that is both throughput/utility optimal and achieves order-optimal delay.

Simulation-based optimization of Markov reward processes

We propose a simulation-based algorithm for optimizing the average reward in a Markov reward process that depends on a set of parameters. As a special case, the method applies to Markov decision processes where optimization takes place within a parametrized set of policies. The algorithm involves the simulation of a single sample path, and can be implemented online. A convergence result (with probability 1) is provided.

Pricing differentiated services networks: bursty traffic

We study the role of pricing in differentiated services (Diff-Serv) networks. We model DiffServ as a priority service, where users are given the freedom to choose the priorities of their traffic, but are charged accordingly. Using a game theoretic framework, we study the case where users choose an allocation of priorities to packets in order to optimize their net benefit. For the case where users with bursty traffic access a single link, we show that there always exists an equilibrium for the corresponding noncooperative game. Furthermore we show that pricing can be used to provide relative QoS guarantees.

Approximate Gradient Methods in Policy-Space Optimization of Markov Reward Processes

We consider a discrete time, finite state Markov reward process that depends on a set of parameters. We start with a brief review of (stochastic) gradient descent methods that tune the parameters in order to optimize the average reward, using a single (possibly simulated) sample path of the process of interest. The resulting algorithms can be implemented online, and have the property that the gradient of the average reward converges to zero with probability 1. On the other hand, the updates can have a high variance, resulting in slow convergence. We address this issue and propose two approaches to reduce the variance. These approaches rely on approximate gradient formulas, which introduce an additional bias into the update direction. We derive bounds for the resulting bias terms and characterize the asymptotic behavior of the resulting algorithms. For one of the approaches considered, the magnitude of the bias term exhibits an interesting dependence on the time it takes for the rewards to reach steady-state. We also apply the methodology to Markov reward processes with a reward-free termination state, and an expected total reward criterion. We use a call admission control problem to illustrate the performance of the proposed algorithms.