Panayotis Mertikopoulos

Distributed Learning Policies for Power Allocation in Multiple Access Channels

We analyze the power allocation problem for orthogonal multiple access channels by means of a non-cooperative potential game in which each user distributes his power over the channels available to him. When the channels are static, we show that this game possesses a unique equilibrium; moreover, if the networks users follow a distributed learning scheme based on the replicator dynamics of evolutionary game theory, then they converge to equilibrium exponentially fast. On the other hand, if the channels fluctuate stochastically over time, the associated game still admits a unique equilibrium, but the learning process is not deterministic; just the same, by employing the theory of stochastic approximation, we find that users still converge to equilibrium. Our theoretical analysis hinges on a novel result which is of independent interest: in finite-player games which admit a (possibly nonlinear) convex potential, the replicator dynamics converge to an e-neighborhood of an equilibrium in time O(\log(1/e)).

Energy-Aware Competitive Power Allocation for Heterogeneous Networks Under QoS Constraints

This work proposes a distributed power allocation scheme for maximizing energy efficiency in the uplink of OFDMA-based HetNets where a macro-tier is augmented with small cell access points. Each user equipment (UE) in the network is modeled as a rational agent that engages in a non-cooperative game and allocates its available transmit power over the set of assigned subcarriers to maximize its individual utility (defined as the users throughput per Watt of transmit power) subject to a target rate requirement. In this framework, the relevant solution concept is that of Debreu equilibrium, a generalization of the concept of Nash equilibrium. Using techniques from fractional programming, we provide a characterization of equilibrial power allocation profiles. In particular, Debreu equilibria are found to be the fixed points of a water-filling best response operator whose water level is a function of rate constraints and circuit power. Moreover, we also describe a set of sufficient conditions for the existence and uniqueness of Debreu equilibria exploiting the contraction properties of the best response operator. This analysis provides the necessary tools to derive a power allocation scheme that steers the network to equilibrium in an iterative and distributed manner without the need for any centralized processing. Numerical simulations are used to validate the analysis and assess the performance of the proposed algorithm as a function of the system parameters.

Penalty-Regulated Dynamics and Robust Learning Procedures in Games

Starting from a heuristic learning scheme for strategic N -person games, we derive a new class of continuous-time learning dynamics consisting of a replicator-like drift adjusted by a penalty term that renders the boundary of the game’s strategy space repelling. These penalty-regulated dynamics are equivalent to players keeping an exponentially discounted aggregate of their ongoing payoffs and then using a smooth best response to pick an action based on these performance scores. Owing to this inherent duality, the proposed dynamics satisfy a variant of the folk theorem of evolutionary game theory and they converge to (arbitrarily precise) approximations of Nash equilibria in potential games. Motivated by applications to traffic engineering, we exploit this duality further to design a discrete-time, payoff-based learning algorithm that retains these convergence properties and only requires players to observe their in-game payoffs. Moreover, the algorithm remains robust in the presence of stochastic perturbations and observation errors, and it does not require any synchronization between players.

Higher order game dynamics

Continuous-time game dynamics are typically first order systems where payoffs determine the growth rate of the playersʼ strategy shares. In this paper, we investigate what happens beyond first order by viewing payoffs as higher order forces of change, specifying e.g. the acceleration of the playersʼ evolution instead of its velocity (a viewpoint which emerges naturally when it comes to aggregating empirical data of past instances of play). To that end, we derive a wide class of higher order game dynamics, generalizing first order imitative dynamics, and, in particular, the replicator dynamics. We show that strictly dominated strategies become extinct in n-th order payoff-monotonic dynamics n orders as fast as in the corresponding first order dynamics; furthermore, in stark contrast to first order, weakly dominated strategies also become extinct for n⩾2. All in all, higher order payoff-monotonic dynamics lead to the elimination of weakly dominated strategies, followed by the iterated deletion of strictly dominated strategies, thus providing a dynamic justification of the well-known epistemic rationalizability process of Dekel and Fudenberg [7]. Finally, we also establish a higher order analogue of the folk theorem of evolutionary game theory, and we show that convergence to strict equilibria in n-th order dynamics is n orders as fast as in first order.

Mixed-Strategy Learning With Continuous Action Sets

Motivated by the recent applications of game-theoretical learning to the design of distributed control systems, we study a class of control problems that can be formulated as potential games with continuous action sets. We propose an actor-critic reinforcement learning algorithm that adapts mixed strategies over continuous action spaces. To analyze the algorithm, we extend the theory of finite-dimensional two-timescale stochastic approximation to a Banach space setting, and prove that the continuous dynamics of the process converge to equilibrium in the case of potential games. These results combine to give a provably-convergent learning algorithm in which players do not need to keep track of the controls selected by other agents.

Learning in an Uncertain World: MIMO Covariance Matrix Optimization With Imperfect Feedback

In this paper, we present a distributed learning algorithm for the optimization of signal covariance matrices in Gaussian multiple-input and multiple-output (MIMO) multiple access channel with imperfect (and possibly delayed) feedback. The algorithm is based on the method of matrix exponential learning (MXL) and it has the same information and computation requirements as distributed water-filling. However, unlike water-filling, the proposed algorithm converges to the systems optimum signal covariance profile even under stochastic uncertainty and imperfect feedback. Moreover, the algorithm also retains its convergence properties in the presence of user update asynchronicities, random delays and/or ergodically changing channel conditions. Our theoretical analysis is complemented by extensive numerical simulations which illustrate the robustness and scalability of MXL in realistic network conditions. In particular, the algorithm retains its convergence speed even for large numbers of users and/or antennas per user.

Learning in games with continuous action sets and unknown payoff functions

This paper examines the convergence of no-regret learning in games with continuous action sets. For concreteness, we focus on learning via “dual averaging”, a widely used class of no-regret learning schemes where players take small steps along their individual payoff gradients and then “mirror” the output back to their action sets. In terms of feedback, we assume that players can only estimate their payoff gradients up to a zero-mean error with bounded variance. To study the convergence of the induced sequence of play, we introduce the notion of variational stability, and we show that stable equilibria are locally attracting with high probability whereas globally stable equilibria are globally attracting with probability 1. We also discuss some applications to mixed-strategy learning in finite games, and we provide explicit estimates of the method’s convergence speed.

On the robustness of learning in games with stochastically perturbed payoff observations

Motivated by the scarcity of accurate payoff feedback in practical applications of game theory, we examine a class of learning dynamics where players adjust their choices based on past payoff observations that are subject to noise and random disturbances. First, in the single-player case (corresponding to an agent trying to adapt to an arbitrarily changing environment), we show that the stochastic dynamics under study lead to no regret almost surely, irrespective of the noise level in the players observations. In the multi-player case, we find that dominated strategies become extinct and we show that strict Nash equilibria are stochastically stable and attracting; conversely, if a state is stable or attracting with positive probability, then it is a Nash equilibrium. Finally, we provide an averaging principle for 2-player games, and we show that in zero-sum games with an interior equilibrium, time averages converge to Nash equilibrium for any noise level.

Auction-based resource allocation in OpenFlow multi-tenant networks

In this paper, we investigate the allocation of network resources (such as FlowTable entries and bandwidth) in multi-tenant Software-Defined Networks (SDNs) that are managed by a FlowVisor. This resource allocation problem is modeled as an auction where the FlowVisor acts as the auctioneer and the network Controllers act as the bidders. The problem is analyzed by means of non-cooperative game theory, and it is shown that the auction admits a unique Nash Equilibrium (NE) under suitable conditions. Furthermore, a novel distributed learning procedure is provided that allows each Controller to reach the games unique NE in a few iterations by exploiting only locally available information. An implementation in OpenFlow-compliant SDNs is also proposed in a way that exploits native procedures already offered by OpenFlow. Finally, simulation results show that the proposed auction-based resource management scheme leads to significant improvements in network performance (for instance, achieving gains of up to 5 reduction in transmission delays).

Energy-aware competitive link adaptation in small-cell networks

This work proposes a distributed power allocation scheme for maximizing the energy efficiency in the uplink of non-cooperative small-cell networks based on orthogonal frequency-division multiple-access technology. This is achieved by modeling user terminals as rational agents that engage in a non-cooperative game in which every terminal selects the power loading so as to maximize its own utility (the users throughput per Watt of transmit power) while satisfying minimum rate constraints. In this framework, we prove the existence of a Debreu equilibrium (also known as generalized Nash equilibrium) and we characterize the structure of the corresponding power allocation profile using techniques drawn from fractional programming. To attain the equilibrium in a distributed fashion, we also propose a method based on an iterative water-filling best response process. Numerical simulations are then used to assess the convergence of the proposed algorithm and the performance of its end-state as a function of the system parameters.