Marc J. V. Ponsen

Adaptive game AI with dynamic scripting

Online learning in commercial computer games allows computer-controlled opponents to adapt to the way the game is being played. As such it provides a mechanism to deal with weaknesses in the game AI, and to respond to changes in human player tactics. We argue that online learning of game AI should meet four computational and four functional requirements. The computational requirements are speed, effectiveness, robustness and efficiency. The functional requirements are clarity, variety, consistency and scalability. This paper investigates a novel online learning technique for game AI called ‘dynamic scripting’, that uses an adaptive rulebase for the generation of game AI on the fly. The performance of dynamic scripting is evaluated in experiments in which adaptive agents are pitted against a collection of manually-designed tactics in a simulated computer roleplaying game. Experimental results indicate that dynamic scripting succeeds in endowing computer-controlled opponents with adaptive performance. To further improve the dynamic-scripting technique, an enhancement is investigated that allows scaling of the difficulty level of the game AI to the human player’s skill level. With the enhancement, dynamic scripting meets all computational and functional requirements. The applicability of dynamic scripting in state-of-the-art commercial games is demonstrated by implementing the technique in the game Neverwinter Nights. We conclude that dynamic scripting can be successfully applied to the online adaptation of game AI in commercial computer games.

Learning to win: case-based plan selection in a real-time strategy game

While several researchers have applied case-based reasoning techniques to games, only Ponsen and Spronck (2004) have addressed the challenging problem of learning to win real-time games. Focusing on Wargus, they report good results for a genetic algorithm that searches in plan space, and for a weighting algorithm (dynamic scripting) that biases subplan retrieval. However, both approaches assume a static opponent, and were not designed to transfer their learned knowledge to opponents with substantially different strategies. We introduce a plan retrieval algorithm that, by using three key sources of domain knowledge, removes the assumption of a static opponent. Our experiments show that its implementation in the Case-based Tactician (CaT) significantly outperforms the best among a set of genetically evolved plans when tested against random Wargus opponents. CaT communicates with Wargus through TIELT, a testbed for integrating and evaluating decision systems with simulators. This is the first application of TIELT. We describe this application, our lessons learned, and our motivations for future work.

Knowledge acquisition for adaptive game AI

Game artificial intelligence (AI) controls the decision-making process of computer-controlled opponents in computer games. Adaptive game AI (i.e., game AI that can automatically adapt the behaviour of the computer players to changes in the environment) can increase the entertainment value of computer games. Successful adaptive game AI is invariably based on the games domain knowledge. We show that an offline evolutionary algorithm can learn important domain knowledge in the form of game tactics (i.e., a sequence of game actions) for dynamic scripting, an offline algorithm inspired by reinforcement learning approaches that we use to create adaptive game AI. We compare the performance of dynamic scripting under three conditions for defeating non-adaptive opponents in a real-time strategy game. In the first condition, we manually encode its tactics. In the second condition, we manually translate the tactics learned by the evolutionary algorithm, and use them for dynamic scripting. In the third condition, this translation is automated. We found that dynamic scripting performs best under the third condition, and both of the latter conditions outperform manual tactic encoding. We discuss the implications of these results, and the performance of dynamic scripting for adaptive game AI from the perspective of machine learning research and commercial game development.

Computing approximate Nash Equilibria and robust best-responses using sampling

This article discusses two contributions to decision-making in complex partially observable stochastic games. First, we apply two state-of-the-art search techniques that use Monte-Carlo sampling to the task of approximating a Nash-Equilibrium (NE) in such games, namely Monte-Carlo Tree Search (MCTS) and Monte-Carlo Counterfactual Regret Minimization (MCCFR). MCTS has been proven to approximate a NE in perfect-information games. We show that the algorithm quickly finds a reasonably strong strategy (but not a NE) in a complex imperfect information game, i.e. Poker. MCCFR on the other hand has theoretical NE convergence guarantees in such a game. We apply MCCFR for the first time in Poker. Based on our experiments, we may conclude that MCTS is a valid approach if one wants to learn reasonably strong strategies fast, whereas MCCFR is the better choice if the quality of the strategy is most important. Our second contribution relates to the observation that a NE is not a best response against players that are not playing a NE. We present Monte-Carlo Restricted Nash Response (MCRNR), a sample-based algorithm for the computation of restricted Nash strategies. These are robust bestresponse strategies that (1) exploit non-NE opponents more than playing a NE and (2) are not (overly) exploitable by other strategies. We combine the advantages of two state-of-the-art algorithms, i.e. MCCFR and Restricted Nash Response (RNR). MCRNR samples only relevant parts of the game tree. We show that MCRNR learns quicker than standard RNR in smaller games. Also we show in Poker that MCRNR learns robust best-response strategies fast, and that these strategies exploit opponents more than playing a NE does.

Effective and Diverse Adaptive Game AI

Adaptive techniques tend to converge to a single optimum. For adaptive game AI, such convergence is often undesirable, as repetitive game AI is considered to be uninteresting for players. In this paper, we propose a method for automatically learning diverse but effective macros that can be used as components of adaptive game AI scripts. Macros are learned by a cross-entropy method (CEM). This is a selection-based optimization method that, in our experiments, maximizes an interestingness measure. We demonstrate the approach in a computer role-playing game (CRPG) simulation with two duelling wizards, one of which is controlled by an adaptive game AI technique called ldquodynamic scripting.rdquo Our results show that the macros that we learned manage to increase both adaptivity and diversity of the scripts generated by dynamic scripting, while retaining playing strength.

An evolutionary game-theoretic analysis of poker strategies

Abstract In this paper we investigate the evolutionary dynamics of strategic behavior in the game of poker by means of data gathered from a large number of real world poker games. We perform this study from an evolutionary game theoretic perspective using two Replicator Dynamics models. First we consider the basic selection model on this data, secondly we use a model which includes both selection and mutation. We investigate the dynamic properties by studying how rational players switch between different strategies under different circumstances, what the basins of attraction of the equilibria look like, and what the stability properties of the attractors are. We illustrate the dynamics using a simplex analysis. Our experimental results confirm existing domain knowledge of the game, namely that certain strategies are clearly inferior while others can be successful given certain game conditions.

Abstraction and generalization in reinforcement learning: a summary and framework

In this paper we survey the basics of reinforcement learning, generalization and abstraction. We start with an introduction to the fundamentals of reinforcement learning and motivate the necessity for generalization and abstraction. Next we summarize the most important techniques available to achieve both generalization and abstraction in reinforcement learning. We discuss basic function approximation techniques and delve into hierarchical, relational and transfer learning. All concepts and techniques are illustrated with examples.