T. E. S. Raghavan

Algorithms for stochastic games — A survey

We consider finite state, finite action, stochastic games over an infinite time horizon. We survey algorithms for the computation of minimax optimal stationary strategies in the zerosum case, and of Nash equilibria in stationary strategies in the nonzerosum case. We also survey those theoretical results that pave the way towards future development of algorithms.ZusammenfassungIn dieser Arbeit werden unendlichstufige stochastische Spiele mit endlichen ZuStands- und Aktionenräumen untersucht. Es wird ein Überblick gegeben über Algorithmen zur Berechnung von optimalen stationären Minimax-Strategien in Nullsummen-Spielen und von stationären Nash-Gleichgewichtsstrategien in Nicht-Nullsummen-Spielen. Einige theoretische Ergebnisse werden vorgestellt, die für die weitere Entwicklung von Algorithmen nützlich sind.

An orderfield property for stochastic games when one player controls transition probabilities

When the transition probabilities of a two-person stochastic game do not depend on the actions of a fixed player at all states, the value exists in stationary strategies. Further, the data of the stochastic game, the values at each state, and the components of a pair of optimal stationary strategies all lie in the same Archimedean ordered field. This orderfield property holds also for the nonzero sum case in Nash equilibrium stationary strategies. A finite-step algorithm for the discounted case is given via linear programming.

Existence of stationary correlated equilibria with symmetric information for discounted stochastic games

The following theorem is proved: Every nonzero-sum discounted stochastic game in countably generated measurable state space with compact metric action spaces admits a stationary correlated equilibrium point with symmetric public information whenever the immediate rewards and transition densities are measurable with respect to the state variable and continuous with respect to joint actions.

Assignment games with stable core

Abstract. We prove that the core of an assignment game (a two-sided matching game with transferable utility as introduced by Shapley and Shubik, 1972) is stable (i.e., it is the unique von Neumann-Morgenstern solution) if and only if there is a matching between the two types of players such that the corresponding entries in the underlying matrix are all row and column maximums. We identify other easily verifiable matrix properties and show their equivalence to various known sufficient conditions for core-stability. By these matrix characterizations we found that on the class of assignment games, largeness of the core, extendability and exactness of the game are all equivalent conditions, and strictly imply the stability of the core. In turn, convexity and subconvexity are equivalent, and strictly imply all aformentioned conditions.

On Stochastic Games with Additive Reward and Transition Structure

In this paper, we introduce a new class of two-person stochastic games with nice properties. For games in this class, the payoffs as well as the transitions in each state consist of a part which depends only on the action of the first player and a part dependent only on the action of the second player.For the zero-sum games in this class, we prove that the orderfield property holds in the infinite-horizon case and that there exist optimal pure stationary strategies for the discounted as well as the undiscounted payoff criterion. For both criteria also, finite algorithms are given to solve the game. An example shows that, for nonzero sum games in this class, there are not necessarily pure stationary equilibria. But, if such a game possesses a stationary equilibrium point, then there also exists a stationary equilibrium point which uses in each state at most two pure actions for each player.

Existence of

In this paper we prove the existence of p-equilibrium stationary strategies for non-zero-sum stochastic games when the reward functions and transitions satisfy certain separability conditions. We also prove some results for positive and discounted zero-sum stochastic games when the state space is infinite.

p

Given a coalition of ann-person cooperative game in characteristic function form, we can associate a zero-one vector whose non-zero coordinates identify the players in the given coalition. The cooperative game with this identification is just a map on such vectors. By allowing each coordinate to take finitely many values we can define multi-choice cooperative games. In such multi-choice games we can also define Shapley value axiomatically. We show that this multi-choice Shapley value is dummy free of actions, dummy free of players, non-decreasing for non-decreasing multi-choice games, and strictly increasing for strictly increasing cooperative games. Some of these properties are closely related to some properties of independent exponentially distributed random variables. An advantage of multi-choice formulation is that it allows to model strategic behavior of players within the context of cooperation.

-equilibrium and optimal stationary strategies in stochastic games

SummaryIn this paper two-person zero-sum stochastic games are considered with the average payoff as criterion. It is assumed that in each state one of the players governs the transitions. We will establish an algorithm, which yields in a finite number of iterations the solution of the game i.e. the value of the game and optimal stationary strategies for both players. An essential part of our algorithm is formed by the linear programming problem which solves a one player control stochastic game. Furthermore, our algorithm provides a constructive proof of the existence of the value and of optimal stationary strategies for both players. In addition, the finiteness of our algorithm proves also the ordered field property of the switching control stochastic game.ZusammenfassungWir betrachten stochastische Zweipersonen-Nullsummenspiele mit der durchschnittlichen Auszahlung als Kriterium. Wir nehmen an, daß in jedem Zustand einer der Spieler das Übergangsgesetz kontrolliert und entwickeln einen Algorithmus, der nach endlichen vielen Iterationsschritten die Lösung des Spiels — d. h. den Spielwert und optimale stationäre Strategien für beide Spieler — liefert. Ein wesentlicher Teil unseres Algorithmus besteht aus dem linearen Programm, das ein stochastisches Spiel löst, bei dem ein Spieler das Übergangsgesetz bestimmt. Darüber hinaus geben wir mit unserem Algorithmus einen konstruktiven Beweis der Existenz des Spielwertes und optimaler stationärer Strategien für beide Spieler. Weiter zeigt die Endlichkeit unseres Algorithmus die “ordered field property” stochastischer Spiele mit wechselnder Kontrolle des Übergangsgesetzes.

Monotonicity and Dummy Free Property for Multi-choice Cooperative Games

The set of correlated equilibria for a bimatrix game is a closed, bounded, convex set containing the set of Nash equilibria. We show that every extreme point of a maximal Nash set is an extreme point of the above convex set. We also give an example to show that this result is not true in the payoff space, i.e. there are games where no Nash equilibrium payoff is an extreme point of the set of correlated equilibrium payoffs.

A finite algorithm for the switching control stochastic game

Given a non-zero sum discounted stochastic game with finitely many states and actions one can form a bimatrix game whose pure strategies are the pure stationary strategies of the players and whose penalty payoffs consist of the total discounted costs over all states at any pure stationary pair. It is shown that any Nash equilibrium point of this bimatrix game can be used to find a Nash equilibrium point of the stochastic game whenever the law of motion is controlled by one player. The theorem is extended to undiscounted stochastic games with irreducible transitions when the law of motion is controlled by one player. Examples are worked out to illustrate the algorithm proposed.