Edward Jack Powley

A Survey of Monte Carlo Tree Search Methods

Monte Carlo tree search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithms derivation, impart some structure on the many variations and enhancements that have been proposed, and summarize the results from the key game and nongame domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.

Information Set Monte Carlo Tree Search

Monte Carlo tree search (MCTS) is an AI technique that has been successfully applied to many deterministic games of perfect information. This paper investigates the application of MCTS methods to games with hidden information and uncertainty. In particular, three new information set MCTS (ISMCTS) algorithms are presented which handle different sources of hidden information and uncertainty in games. Instead of searching minimax trees of game states, the ISMCTS algorithms search trees of information sets, more directly analyzing the true structure of the game. These algorithms are tested in three domains with different characteristics, and it is demonstrated that our new algorithms outperform existing approaches to handling hidden information and uncertainty in games.

Solving the Physical Traveling Salesman Problem: Tree Search and Macro Actions

This paper presents a number of approaches for solving a real-time game consisting of a ship that must visit a number of waypoints scattered around a 2-D maze full of obstacles. The game, the Physical Traveling Salesman Problem (PTSP), which featured in two IEEE conference competitions during 2012, provides a good balance between long-term planning (finding the optimal sequence of waypoints to visit), and short-term planning (driving the ship in the maze). This paper focuses on the algorithm that won both PTSP competitions: it takes advantage of the physics of the game to calculate the optimal order of waypoints, and it employs Monte Carlo tree search (MCTS) to drive the ship. The algorithm uses repetitions of actions (macro actions) to reduce the search space for navigation. Variations of this algorithm are presented and analyzed, in order to understand the strength of each one of its constituents and to comprehend what makes such an approach the best controller found so far for the PTSP.

Monte Carlo Tree Search with macro-actions and heuristic route planning for the Physical Travelling Salesman Problem

We present a controller for the Physical Travelling Salesman Problem (PTSP), a path planning and steering problem in a simulated continuous real-time domain. Our approach is hierarchical, using domain-specific algorithms and heuristics to plan a coarse-grained route and Monte Carlo Tree Search (MCTS) to plan and steer along fine-grained paths. The MCTS component uses macro-actions to decrease the number of decisions to be made per unit of time and thus drastically reduce the size of the decision tree. Results from the 2012 WCCI PTSP Competition show that this approach significantly and consistently outperforms all other submitted AI controllers, and is competitive with strong human players. Our approach has potential applications to many other problems in movement planning and control, including video games.

Ensemble Determinization in Monte Carlo Tree Search for the Imperfect Information Card Game Magic: The Gathering

In this paper, we examine the use of Monte Carlo tree search (MCTS) for a variant of one of the most popular and profitable games in the world: the card game Magic: The Gathering (M:TG). The game tree for M:TG has a range of distinctive features, which we discuss here; it has incomplete information through the opponents hidden cards and randomness through card drawing from a shuffled deck. We investigate a wide range of approaches that use determinization, where all hidden and random information is assumed known to all players, alongside MCTS. We consider a number of variations to the rollout strategy using a range of levels of sophistication and expert knowledge, and decaying reward to encourage play urgency. We examine the effect of utilizing various pruning strategies in order to increase the information gained from each determinization, alongside methods that increase the relevance of random choices. Additionally, we deconstruct the move generation procedure into a binary yes/no decision tree and apply MCTS to this finer grained decision process. We compare our modifications to a basic MCTS approach for M:TG using fixed decks, and show that significant improvements in playing strength can be obtained.

Bandits all the way down: UCB1 as a simulation policy in Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) is a family of asymmetric anytime aheuristic game tree search algorithms which have advanced the state-of-the-art in several challenging domains. MCTS learns a playout policy, iteratively building a partial tree to store and further refine the learned portion of the policy. When the playout leaves the existing tree, it falls back to a default simulation policy, which for many variants of MCTS chooses actions uniformly at random. This paper investigates how a simulation policy can be learned during the search, helping the playout policy remain plausible from root to terminal state without the injection of prior knowledge. Since the simulation policy visits states that are previously unseen, its decisions cannot be as context sensitive as those in the tree policy. We consider the well-known Move-Average Sampling Technique (MAST), which learns a value for each move which is independent of context. We also introduce a generalisation of MAST, called N-gram-Average-Sampling-Technique (NAST), which uses as context a fixed-lengthsequence (or N-tuple) of recent moves. We compare several policies for selecting moves during simulation, including the UCB1 policy for multi-armed bandits (as used in the tree policy for the popular UCT variant of MCTS). In addition to the elegance of treating the entire playout as a series of multi-armed bandit problems, we find that UCB1 gives consistently strong performance. We present empirical results for three games of imperfect information, namely the card games Dou Di Zhu and Hearts and the board game Lord Of The Rings: The Confrontation, each of which has its own unique challenges for search-based AI.

Monte Carlo Tree Search with macro-actions and heuristic route planning for the Multiobjective Physical Travelling Salesman Problem

This paper describes our entry to the Multiobjective Physical Travelling Salesman Problem (MO-PTSP) competition at the IEEE CIG 2013 conference. MO-PTSP combines the classical Travelling Salesman Problem with the task of steering a simulated spaceship on the 2-D plane, requiring that the controller minimises the three objectives of time taken, fuel consumed and damage incurred. Our entry to the MO-PTSP competition builds upon our winning entry to the previous (single-objective) PTSP competitions. This controller consists of two key components: a pre-planning stage using a classical TSP solver with a path cost measure that takes the physics of the problem into account, and a steering controller using Monte Carlo Tree Search (MCTS) with macro-actions (repeated actions), depth limiting and a heuristic fitness function for nonterminal states. We demonstrate that by modifying the two fitness functions we can produce effective behaviour in MO-PTSP without the need for major modifications to the overall architecture. The fitness functions used by our controller have several parameters, which must be set to ensure the best performance. Given the number of parameters and the difficulty of optimising a controller to satisfy multiple objectives in a search space which is many orders of magnitude larger than that encountered in a turn-based game such as Go, we show that informed hand tuning of parameters is insufficient for this task. We present an automatic parameter tuning method using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) algorithm, which produced parameter settings that dominate our hand tuned parameters. Additionally we show that the robustness of the controller using hand tuned parameters can be improved by detecting when the controller is trapped in a poor quality local optimum and escaping by switching to an alternate fitness function.

Information capture and reuse strategies in Monte Carlo Tree Search, with applications to games of hidden information

Monte Carlo Tree Search (MCTS) has produced many breakthroughs in search-based decision-making in games and other domains. There exist many general-purpose enhancements for MCTS, which improve its efficiency and effectiveness by learning information from one part of the search space and using it to guide the search in other parts. We introduce the Information Capture And ReUse Strategy (ICARUS) framework for describing and combining such enhancements. We demonstrate the ICARUS frameworks usefulness as a frame of reference for understanding existing enhancements, combining them, and designing new ones.We also use ICARUS to adapt some well-known MCTS enhancements (originally designed for games of perfect information) to handle information asymmetry between players and randomness, features which can make decision-making much more difficult. We also introduce a new enhancement designed within the ICARUS framework, EPisodic Information Capture and reuse (EPIC), designed to exploit the episodic nature of many games. Empirically we demonstrate that EPIC is stronger and more robust than existing enhancements in a variety of game domains, thus validating ICARUS as a powerful tool for enhancement design within MCTS.

Player Preference and Style in a Leading Mobile Card Game

Tuning game difficulty prior to release requires careful consideration. Players can quickly lose interest in a game if it is too hard or too easy. Assessing how players will cope prior to release is often inaccurate. However, modern games can now collect sufficient data to perform large scale analysis post deployment and update the product based on these insights. AI Factory Spades is currently the top rated Spades game in the Google Play store. In collaboration with the developers, we have collected gameplay data from 27 592 games and statistics regarding wins/losses for 99 866 games using Google Analytics. Using the data collected, this study analyses the difficulty and behavior of an Information Set Monte Carlo Tree Search player we developed and deployed in the game previously. The methods of data collection and analysis presented in this study are generally applicable. The same workflow could be used to analyze the difficulty and typical player or opponent behavior in any game. Furthermore, addressing issues of difficulty or nonhuman-like opponents postdeployment can positively affect player retention.

Heuristic move pruning in Monte Carlo Tree Search for the strategic card game Lords of War

Move pruning is a technique used in game tree search which incorporates heuristic knowledge to reduce the number of moves under consideration from a particular game state. This paper investigates Heuristic Move Pruning on the strategic card game Lords of War. We use heuristics to guide our pruning and experiment with different techniques of applying pruning and their relative effectiveness. We also present a technique of artificially rolling forward a game state in an attempt to more accurately determine which moves are appropriate to prune from the decision tree. We demonstrate that heuristic move pruning is effective in Lords of War, and also that artificially rolling forward the game state can increase the effectiveness of heuristic move pruning.