Chris J. Maddison

Mastering the game of Go with deep neural networks and tree search

The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of state-of-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.

Rapid and Widespread Effects of 17β-Estradiol on Intracellular Signaling in the Male Songbird Brain: A Seasonal Comparison

Across vertebrate species, 17β-estradiol (E(2)) acts on the brain via both genomic and nongenomic mechanisms to influence neuronal physiology and behavior. Nongenomic E(2) signaling is typically initiated by membrane-associated estrogen receptors that modulate intracellular signaling cascades, including rapid phosphorylation of ERK. Phosphorylated ERK (pERK) can, in turn, rapidly phosphorylate tyrosine hydroxylase (TH) and cAMP response element-binding protein (CREB). Recent data suggest that the rapid effects of E(2) on mouse aggressive behavior are more prominent during short photoperiods (winter) and that acute aromatase inhibition reduces songbird aggression in winter only. To date, seasonal plasticity in the rapid effects of E(2) on intracellular signaling has not been investigated. Here, we compared the effects of acute (15 min) E(2) treatment on pERK, pTH, and pCREB immunoreactivity in male song sparrows (Melospiza melodia) pretreated with the aromatase inhibitor fadrozole during the breeding and nonbreeding seasons. We examined immunoreactivity in 14 brain regions including portions of the song control system, social behavior network, and the hippocampus (Hp). In both seasons, E(2) significantly decreased pERK in nucleus taeniae of the amygdala, pTH in ventromedial hypothalamus, and pCREB in mesencephalic central gray, robust nucleus of the arcopallium, and caudomedial nidopallium. However, several effects were critically dependent upon season. E(2) decreased pERK in caudomedial nidopallium in the breeding season only and decreased pCREB in the medial preoptic nucleus in the nonbreeding season only. Remarkably, E(2) decreased pERK in Hp in the breeding season but increased pERK in Hp in the nonbreeding season. Together, these data demonstrate that E(2) has rapid effects on intracellular signaling in multiple regions of the male brain and also demonstrate that rapid effects of E(2) can be profoundly different across the seasons.

Annealing between distributions by averaging moments

Many powerful Monte Carlo techniques for estimating partition functions, such as annealed importance sampling (AIS), are based on sampling from a sequence of intermediate distributions which interpolate between a tractable initial distribution and the intractable target distribution. The near-universal practice is to use geometric averages of the initial and target distributions, but alternative paths can perform substantially better. We present a novel sequence of intermediate distributions for exponential families defined by averaging the moments of the initial and target distributions. We analyze the asymptotic performance of both the geometric and moment averages paths and derive an asymptotically optimal piecewise linear schedule. AIS with moment averaging performs well empirically at estimating partition functions of restricted Boltzmann machines (RBMs), which form the building blocks of many deep learning models.