Guillaume Chaslot

PROGRESSIVE STRATEGIES FOR MONTE-CARLO TREE SEARCH

Monte-Carlo Tree Search (MCTS) is a new best-first search guided by the results of Monte-Carlo simulations. In this article, we introduce two progressive strategies for MCTS, called progressive bias and progressive unpruning. They enable the use of relatively time-expensive heuristic knowledge without speed reduction. Progressive bias directs the search according to heuristic knowledge. Progressive unpruning first reduces the branching factor, and then increases it gradually again. Experiments assess that the two progressive strategies significantly improve the level of our Go program Mango. Moreover, we see that the combination of both strategies performs even better on larger board sizes.

Parallel Monte-Carlo Tree Search

Monte-Carlo Tree Search (MCTS) is a new best-first search method that started a revolution in the field of Computer Go. Parallelizing MCTS is an important way to increase the strength of any Go program. In this article, we discuss three parallelization methods for MCTS: leaf parallelization, root parallelization, and tree parallelization. To be effective tree parallelization requires two techniques: adequately handling of (1) local mutexesand (2) virtual loss. Experiments in 13×13 Go reveal that in the program Mango root parallelization may lead to the best results for a specific time setting and specific program parameters. However, as soon as the selection mechanism is able to handle more adequately the balance of exploitation and exploration, tree parallelization should have attention too and could become a second choice for parallelizing MCTS. Preliminary experiments on the smaller 9×9 board provide promising prospects for tree parallelization.

Single-Player Monte-Carlo Tree Search

Classical methods such as A* and IDA* are a popular and successful choice for one-player games. However, they fail without an accurate admissible evaluation function. In this paper we investigate whether Monte-Carlo Tree Search (MCTS) is an interesting alternative for one-player games where A* and IDA* methods do not perform well. Therefore, we propose a new MCTS variant, called Single-Player Monte-Carlo Tree Search (SP-MCTS). The selection and backpropagation strategy in SP-MCTS are different from standard MCTS. Moreover, SP-MCTS makes use of a straightforward Meta-Search extension. We tested the method on the puzzle SameGame. It turned out that our SP-MCTS program gained the highest score so far on the standardized test set.

Adding expert knowledge and exploration in monte-carlo tree search

We present a new exploration term, more efficient than classical UCT-like exploration terms. It combines efficiently expert rules, patterns extracted from datasets, All-Moves-As-First values, and classical online values. As this improved bandit formula does not solve several important situations (semeais, nakade) in computer Go, we present three other important improvements which are central in the recent progress of our program MoGo. We show an expert-based improvement of Monte-Carlo simulations for nakade situations; we also emphasize some limitations of this modification. We show a technique which preserves diversity in the Monte-Carlo simulation, which greatly improves the results in 19x19. Whereas the UCB-based exploration term is not efficient in MoGo, we show a new exploration term which is highly efficient in MoGo. MoGo recently won a game with handicap 7 against a 9Dan Pro player, Zhou JunXun, winner of the LG Cup 2007, and a game with handicap 6 against a 1Dan pro player, Li-Chen Chien.

Monte-Carlo Go Reinforcement Learning Experiments

This paper describes experiments using reinforcement learning techniques to compute pattern urgencies used during simulations performed in a Monte-Carlo Go architecture. Currently, Monte-Carlo is a popular technique for computer Go. In a previous study, Monte-Carlo was associated with domain-dependent knowledge in the Go-playing program Indigo. In 2003, a 3times3 pattern database was built manually. This paper explores the possibility of using reinforcement learning to automatically tune the 3times3 pattern urgencies. On 9times9 boards, within the Monte-Carlo architecture of Indigo, the result obtained by our automatic learning experiments is better than the manual method by a 3-point margin on average, which is satisfactory. Although the current results are promising on 19times19 boards, obtaining strictly positive results with such a large size remains to be done

Monte-Carlo tree search in settlers of catan

Games are considered important benchmark opportunities for artificial intelligence research. Modern strategic board games can typically be played by three or more people, which makes them suitable test beds for investigating multi-player strategic decision making. Monte-Carlo Tree Search (MCTS) is a recently published family of algorithms that achieved successful results with classical, two-player, perfect-information games such as Go. In this paper we apply MCTS to the multi-player, non-deterministic board game Settlers of Catan. We implemented an agent that is able to play against computer-controlled and human players. We show that MCTS can be adapted successfully to multi-agent environments, and present two approaches of providing the agent with a limited amount of domain knowledge. Our results show that the agent has a considerable playing strength when compared to game implementation with existing heuristics. So, we may conclude that MCTS is a suitable tool for achieving a strong Settlers of Catan player.

An Ontology-based Fuzzy Inference System for Computer Go Applications

In order to stimulate the development and research in computer Go, several Taiwanese Go players were invited to play against some famous computer Go programs. Those competitions revealed that the ontology model for Go game might resolve problems happened in the competitions. Therefore, this paper presents a Go game record ontology and Go board ontology schemes. An ontology-based fuzzy inference system is also developed to provide the regional alarm level for a Go beginner or a computer Go program in order to place the stone at the much more appropriate position. Experimental results indicate that the proposed approach is feasible for computer Go application. Hopefully, advances in the intelligent agent and the ontology model can provide a significant amount of knowledge to make a progress in computer Go program and achieve as much as computer chess or Chinese chess in the future.

A novel ontology for computer go knowledge management

In order to stimulate the development and research in computer Go, several Taiwanese Go players, including three professional Go players and four amateur Go players, were invited to play against the famous computer Go program, MoGo, in the Taiwan Open 2009. The MoGo program combines the online game values, offline values extracted from databases, and expert rules defined by Go expert that shows an excellent performance in the games. The results reveal that MoGo can reach the level of 3 Dan in Taiwan amateur Go environment. But there are still some drawbacks for MoGo that should be solved, for example, the weaknesses in semeai and how to flexibly practice the human knowledge through the embedded opening books. In this paper, a new game record ontology for computer Go knowledge management is proposed to solve the problems that MoGo is facing. It is hoped that the advances in intelligent agent and ontology model can provide much more knowledge to make a progress in computer Go and achieve as much as computer chess or Chinese chess in the future.

Monte-Carlo proof-number search for computer Go

In the last decade, proof-number search and Monte-Carlo methods have successfully been applied to the combinatorial-games domain. Proof-number search is a reliable algorithm. It requires a well defined goal to prove. This can be seen as a disadvantage. In contrast to proof-number search, Monte-Carlo evaluation is a flexible stochastic evaluation for game-tree search. In order to improve the efficiency of proof-number search, we introduce a new algorithm, Monte-Carlo Proof-Number search. It enhances proof-number search by adding the flexible Monte-Carlo evaluation. We present the new algorithm and evaluate it on a sub-problem of Go, the Life-and-Death problem. The results show a clear improvement in time efficiency and memory usage: the test problems are solved two times faster and four times less nodes are expanded on average. Future work will assess possibilities to extend this method to other enhanced Proof-Number techniques.

Combiner connaissances expertes, hors-ligne, transientes et en ligne pour l'exploration Monte-Carlo Apprentissage et MC

We combine for Monte-Carlo exploration machine learning at four different time scales: - online regret, through the use of bandit algorithms and Monte-Carlo estimates; - transient learning, through the use of rapid action value estimates (RA VE) which are learnt online and used for accelerating the exploration and are thereafter neglected; - offline learning, by data mining ofdatasets of games; - use of expert knowledge coming from the old ages as prior information. The resulting algorithm is stronger than each element separately. We finally emphasize the exploration-exploitation dilemna in the Monte-Carlo simulations and show great improvements that can be reached with a fine tuning of related constants.