Tomoyuki Kaneko

Large-scale optimization for evaluation functions with minimax search

This paper presents a new method, Minimax Tree Optimization (MMTO), to learn a heuristic evaluation function of a practical alpha-beta search program. The evaluation function may be a linear or non-linear combination of weighted features, and the weights are the parameters to be optimized. To control the search results so that the move decisions agree with the game records of human experts, a well-modeled objective function to be minimized is designed. Moreover, a numerical iterative method is used to find local minima of the objective function, and more than forty million parameters are adjusted by using a small number of hyper parameters. This method was applied to shogi, a major variant of chess in which the evaluation function must handle a larger state space than in chess. Experimental results show that the large-scale optimization of the evaluation function improves the playing strength of shogi programs, and the new method performs significantly better than other methods. Implementation of the new method in our shogi program Bonanza made substantial contributions to the programs first-place finish in the 2013 World Computer Shogi Championship. Additionally, we present preliminary evidence of broader applicability of our method to other two-player games such as chess.

Evaluation of Game Tree Search Methods by Game Records

This paper presents a method of evaluating game tree search methods including standard min-max search with heuristic evaluation functions and Monte Carlo tree search, which recently achieved drastic improvements in the strength of Computer Go programs. The basic idea of this paper is to use an averaged win probability of positions having similar evaluation values. Accuracy measures of evaluation values with respect to win probabilities can be used to assess the performance of game tree search methods. A plot of win probabilities against evaluation values should have consistency and monotonicity if the evaluation values are produced by a good game tree search method. By inspecting whether the plot has the properties for some subset of positions, we can detect specific deficiencies in the game tree search method. We applied our method to Go, Shogi, and Chess, and by comparing the results with empirical understanding of the performance of various game tree search methods and with the results of self-plays, we show that our method is efficient and effective.

Analysis of Evaluation-Function Learning by Comparison of Sibling Nodes

This paper discusses gradients of search values with a parameter vector θ in an evaluation function. Recent learning methods for evaluation functions in computer shogi are based on minimization of an objective function with search results. The gradients of the evaluation function at the leaf position of a principal variation (PV) are used to make an easy substitution of the gradients of the search result. By analyzing the variations of the min-max value, we show (1) when the min-max value is partially differentiable and (2) how the substitution may introduce errors. Experiments on a shogi program with about a million parameters show how frequently such errors occur, as well as how effective the substitutions for parameter tuning are in practice.

Evaluation of Monte Carlo tree search and the application to Go

Recent improvements to Monte Carlo tree search have produced strong computer Go programs. This paper presents a method of measuring the accuracy of Monte Carlo tree search in game programming. We use the win percentage of positions in a large database of game records as a benchmark and compare the win probability obtained by simulations with the benchmark. By applying our method to Monte Carlo tree search in Go, we found differences between search methods and their parameters, and the effect of the properties of positions such as the move numbers and the existence of stones in threats. This paper also introduces numerical metrics to evaluate the performance of search methods. Our experiments in Go, as well as Chess, Othello, and Shogi revealed that the metrics were quite close to our empirical understanding of the performance of various search methods and their parameters.

Automated Identification of Patterns in Evaluation Functions

This paper proposes a general and automated method that generates accurate evaluation functions, without expert players’ knowledge of a target game. Patterns (which are partial descriptions of a game state) are widely used as primitives of evaluation functions in game programming. They have to be carefully selected in order to generate accurate evaluation functions. Our approach consists of three steps: (1) generation of logic formulae by using the specifications of a target game, (2) translation of the formulae into patterns, and (3) selection of a set of suitable patterns from those generated. The problem, in the automated identification of suitable patterns, is that it is difficult either to generate only useful patterns or to examine all possible patterns. The latter obstacle is due to the prohibitive numbers involved. We solved this dilemma by a combination of two methods, where one method generates patterns of good quality, and the other method entails a lightweight selection based on statistics that could handle a large number of candidates. Experiments in Othello revealed that about 100,000 patterns from more than eight million automatically generated patterns could be successfully selected with our method, and that accurate evaluation functions were constructed. This accuracy is comparable to that of specialized Othello programs and is much better than that of the evaluation functions generated by existing general methods.

Enhancements in Monte Carlo tree search algorithms for biased game trees

Monte Carlo tree search (MCTS) algorithms have been applied to various domains and achieved remarkable success. However, it is relatively unclear what game properties enhance or degrade the performance of MCTS, while the largeness of search space including pruning efficiency mainly governs the performance of classical minimax search, assuming a decent evaluation function is given. Existing research has shown that the distribution of suboptimal moves and the non-uniformity of tree shape are more important than the largeness of state space in discussing the performance of MCTS. Our study showed that another property, bias in suboptimal moves, is also important, and we present an enhancement to better handle such situations. We focus on a game tree in which the game-theoretical value is even, while suboptimal moves for a player tend to contain more inferior moves than those for the opponent. We conducted experiments on a standard incremental tree model with various MCTS algorithms based on UCB1, KL-UCB, or Thompson sampling. The results showed that the bias in suboptimal moves degraded the performance of all algorithms and that our enhancement alleviated the effect caused by this property.

LinUCB Applied to Monte-Carlo Tree Search

UCT is a de facto standard method for Monte-Carlo tree search (MCTS) algorithms, which have been applied to various domains and have achieved remarkable success. This study proposes a family of LinUCT algorithms that incorporate LinUCB into MCTS algorithms. LinUCB is a recently developed method that generalizes past episodes by ridge regression with feature vectors and rewards. LinUCB outperforms UCB1 in contextual multi-armed bandit problems. We introduce a straightforward application of LinUCB, (text {LinUCT}_{text {PLAIN}}) by substituting UCB1 with LinUCB in UCT. We show that it does not work well owing to the minimax structure of game trees. To better handle such tree structures, we present (text {LinUCT}_{text {RAVE}}) and (text {LinUCT}_{text {FP}}) by further incorporating two existing techniques, rapid action value estimation (RAVE) and feature propagation, which recursively propagates the feature vector of a node to that of its parent. Experiments were conducted with a synthetic model, which is an extension of the standard incremental random tree model in which each node has a feature vector that represents the characteristics of the corresponding position. The experimental results indicate that (text {LinUCT}_{text {RAVE}}), (text {LinUCT}_{text {FP}}), and their combination (text {LinUCT}_{text {RAVE-FP}}) outperform UCT, especially when the branching factor is relatively large.

A System-Design Outline of the Distributed-Shogi-System Akara 2010

This paper describes Akara 2010, the distributed shogi system that has defeated a professional shogi player in a public game for the first time in history. The system employs a novel design to build a high-performance computer shogi player for standard tournament conditions. The design enhances the performance of the entire system by means of distributed computing. To utilize a large number of computers, a majority-voting method using four existing programs is combined with a distributed-search method. Although the performance of the entire system could not be tested, the majority-voting component increased the winning percentage from 62% to 73%, and the distributed-search component increased it from 50% to 70% or more.

Infinite Connect-Four Is Solved: Draw

In this paper, we present the newly obtained solution for variants of Connect-Four played on an infinite board. We proved this result by introducing never-losing strategies for both players. The strategies consist of a combination of paving patterns, which are follow-up, follow-in-CUP, and a few others. By employing the strategies, both players can block their opponents to achieve the winning condition. This means that optimal play by both players leads to a draw in these games.

Compiling logical features into specialized state-evaluators by partial evaluation, boolean tables and incremental calculation

A good evaluation function is needed for a good game program, and good features, which are primitive metrics of a state, are needed for a good evaluation function. In order to obtain good features, automatic generation of features by machine learning is promising. However, the generated features are usually written in logic programs, whose evaluation is much slower than that of other native expressions due to the interpretive evaluation of the logic programs. In order to solve this problem, we propose a method which constructs a specialized evaluator using a combination of techniques: partial evaluation, Boolean tables, and incremental calculation. It exhaustively unfolds logical programs until they can be represented as simple Boolean tables. The constructed specialized evaluator is elRcient since it consults only these compiled tables. Experiments with Othello showed that speed can be increased approximately 2,000 times.