Jerzy A. Filar

Algorithms for stochastic games — A survey

We consider finite state, finite action, stochastic games over an infinite time horizon. We survey algorithms for the computation of minimax optimal stationary strategies in the zerosum case, and of Nash equilibria in stationary strategies in the nonzerosum case. We also survey those theoretical results that pave the way towards future development of algorithms.ZusammenfassungIn dieser Arbeit werden unendlichstufige stochastische Spiele mit endlichen ZuStands- und Aktionenräumen untersucht. Es wird ein Überblick gegeben über Algorithmen zur Berechnung von optimalen stationären Minimax-Strategien in Nullsummen-Spielen und von stationären Nash-Gleichgewichtsstrategien in Nicht-Nullsummen-Spielen. Einige theoretische Ergebnisse werden vorgestellt, die für die weitere Entwicklung von Algorithmen nützlich sind.

Variance-penalized Markov decision processes

We consider a Markov decision process with both the expected limiting average, and the discounted total return criteria, appropriately modified to include a penalty for the variability in the stream of rewards. In both cases we formulate appropriate nonlinear programs in the space of state-action frequencies averaged, or discounted whose optimal solutions are shown to be related to the optimal policies in the corresponding “variance-penalized MDP.” The analysis of one of the discounted cases is facilitated by the introduction of a “Cartesian product of two independent MDPs.”

Analytic Perturbation Theory and Its Applications

Mathematical models are often used to describe complex phenomena such as climate change dynamics, stock market fluctuations, and the Internet. These models typically depend on estimated values of key parameters that determine system behavior. Hence it is important to know what happens when these values are changed. The study of single-parameter deviations provides a natural starting point for this analysis in many special settings in the sciences, engineering, and economics. The difference between the actual and nominal values of the perturbation parameter is small but unknown, and it is important to understand the asymptotic behavior of the system as the perturbation tends to zero. This is particularly true in applications with an apparent discontinuity in the limiting behavior - the so-called singularly perturbed problems. Analytic Perturbation Theory and Its Applications includes a comprehensive treatment of analytic perturbations of matrices, linear operators, and polynomial systems, particularly the singular perturbation of inverses and generalized inverses. It also offers original applications in Markov chains, Markov decision processes, optimization, and applications to Google PageRank and the Hamiltonian cycle problem as well as input retrieval in linear control systems and a problem section in every chapter to aid in course preparation. Audience: This text is appropriate for mathematicians and engineers interested in systems and control. It is also suitable for advanced undergraduate, first-year graduate, and advanced, one-semester, graduate classes covering perturbation theory in various mathematical areas. Contents: Chapter 1: Introduction and Motivation; Part I: Finite Dimensional Perturbations; Chapter 2: Inversion of Analytically Perturbed Matrices; Chapter 3: Perturbation of Null Spaces, Eigenvectors, and Generalized Inverses; Chapter 4: Polynomial Perturbation of Algebraic Nonlinear Systems; Part II: Applications to Optimization and Markov Process; Chapter 5: Applications to Optimization; Chapter 6: Applications to Markov Chains; Chapter 7: Applications to Markov Decision Processes; Part III: Infinite Dimensional Perturbations; Chapter 8: Analytic Perturbation of Linear Operators; Chapter 9: Background on Hilbert Spaces and Fourier Analysis; Bibliography; Index

Percentile performance criteria for limiting average Markov decision processes

Addresses the following basic feasibility problem for infinite-horizon Markov decision processes (MDPs): can a policy be found that achieves a specified value (target) of the long-run limiting average reward at a specified probability level (percentile)? Related optimization problems of maximizing the target for a specified percentile and vice versa are also considered. The authors present a complete (and discrete) classification of both the maximal achievable target levels and of their corresponding percentiles. The authors also provide an algorithm for computing a deterministic policy corresponding to any feasible target-percentile pair. Next the authors consider similar problems for an MDP with multiple rewards and/or constraints. This case presents some difficulties and leads to several open problems. An LP-based formulation provides constructive solutions for most cases. >

How Airlines and Airports Recover from Schedule Perturbations: A Survey

The explosive growth in air traffic as well as the widespread adoption of Operations Research techniques in airline scheduling has given rise to tight flight schedules at major airports. An undesirable consequence of this is that a minor incident such as a delay in the arrival of a small number of flights can result in a chain reaction of events involving several flights and airports, causing disruption throughout the system. This paper reviews recent literature in the area of recovery from schedule disruptions. First we review how disturbances at a given airport could be handled, including the effects of runways and fixes. Then we study the papers on recovery from airline schedule perturbations, which involve adjustments in flight schedules, aircraft, and crew. The mathematical programming techniques used in ground holding are covered in some detail. We conclude the review with suggestions on how singular perturbation theory could play a role in analyzing disruptions to such highly sensitive schedules as those in the civil aviation industry.

Perturbation and stability theory for Markov control problems

A unified approach to the asymptotic analysis of a Markov decision process disturbed by an epsilon -additive perturbation is proposed. Irrespective of whether the perturbation is regular or singular, the underlying control problem that needs to be understood is the limit Markov control problem. The properties of this problem are studied. >

Control and Game Theoretic Models of the Environment

This work focuses on the application of dynamic game theory and control theory to analyze a number of important issues in the interaction between economic and environmental problems. The authors provide research results, mathematical models and prescriptions for public policy tools to prevent economic and natural systems from taking unsustainable development paths. The book should be of interest to research workers in industry and university, including engineers, mathematicians, economists and public policy-makers.

Nonlinear programming and stationary equilibria in stochastic games

Stationary equilibria in discounted and limiting average finite state/action space stochastic games are shown to be equivalent to global optima of certain nonlinear programs. For zero sum limiting average games, this formulation reduces to a linear objective, nonlinear constraints program, which finds the “best” stationary strategies, even whenε-optimal stationary strategies do not exist, for arbitrarily smallε.

Ordered field property for stochastic games when the player who controls transitions changes from state to state

In this paper, we consider a zero-sum stochastic game with finitely many states restricted by the assumption that the probability transitions from a given state are functions of the actions of only one of the players. However, the player who thus controls the transitions in the given state will not be the same in every state. Further, we assume that all payoffs and all transition probabilities specifying the law of motion are rational numbers. We then show that the values of both a β-discounted game, for rational β, and of a Cesaro-average game are in the field of rational numbers. In addition, both games possess optimal stationary strategies which have only rational components. Our results and their proofs form an extension of the results and techniques which were recently developed by Parthasarathy and Raghavan (Ref. 1).

Singular Perturbations of Markov Chains and Decision Processes

In this survey we present a unified treatment of both singular and regular perturbations in finite Markov chains and decision processes. The treatment is based on the analysis of series expansions of various important entities such as the perturbed stationary distribution matrix, the deviation matrix, the mean-passage times matrix and others.