Jacob Schrum

UT⁁2: Human-like behavior via neuroevolution of combat behavior and replay of human traces

The UT⁁2 bot, which had a humanness rating of 27.2727% in BotPrize 2010, is based on two core ideas: (1) multiobjective neuroevolution is used to learn skilled combat behavior, but filters on the available combat actions ensure that the behavior is still human-like despite being evolved for performance, and (2) a database of traces of human play is used to help the bot get unstuck when its navigation capabilities fail. Several changes have recently been made to UT⁁2: Extra input features have been provided to the bot to help it evolve better combat behavior, the role of human traces in the navigation of the bot has been expanded, and an extra control module has been added which encourages the bot to observe other players the way a human would, rather than simply battle them. These changes should make UT⁁2 act more human-like in this years BotPrize competition.

Evolving Multimodal Networks for Multitask Games

Intelligent opponent behavior makes video games interesting to human players. Evolutionary computation can discover such behavior, however, it is challenging to evolve behavior that consists of multiple separate tasks. This paper evaluates three ways of meeting this challenge via neuroevolution: 1) multinetwork learns separate controllers for each task, which are then combined manually; 2) multitask evolves separate output units for each task, but shares information within the networks hidden layer; and 3) mode mutation evolves new output modes, and includes a way to arbitrate between them. Whereas the fist two methods require that the task division be known, mode mutation does not. Results in Front/Back Ramming and Predator/Prey games show that each of these methods has different strengths. Multinetwork is good in both domains, taking advantage of the clear division between tasks. Multitask performs well in Front/Back Ramming, in which the relative difficulty of the tasks is even, but poorly in Predator/Prey, in which it is lopsided. Interestingly, mode mutation adapts to this asymmetry and performs well in Predator/Prey. This result demonstrates how a human-specified task division is not always the best. Altogether the results suggest how human knowledge and learning can be combined most effectively to evolve multimodal behavior.

Evolving multimodal behavior with modular neural networks in Ms. Pac-Man

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required to succeed: Ms. Pac-Man must escape ghosts when they are threats, and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multiobjective NEAT to evolve modular neural networks. Each module defines a separate policy; evolution discovers these policies and when to use them. The number of modules can be fixed or learned using a new version of a genetic operator, called Module Mutation, which duplicates an existing module that can then evolve to take on a distinct behavioral identity. Both the fixed modular networks and Module Mutation networks outperform traditional monolithic networks. More interestingly, the best modular networks dedicate modules to critical behaviors that do not follow the customary division of the game into chasing edible and escaping threatening ghosts.

Human-Like Combat Behaviour via Multiobjective Neuroevolution

Although evolution has proven to be a powerful search method for discovering effective behaviour for sequential decision-making problems, it seems unlikely that evolving for raw performance could result in behaviour that is distinctly human-like. This chapter demonstrates how human-like behaviour can be evolved by restricting a bot’s actions in a way consistent with human limitations and predilections. This approach evolves good behaviour, but assures that it is consistent with how humans behave. The approach is demonstrated in the \({UT{\char 94}2}\) bot for the commercial first-person shooter videogame Unreal Tournament 2004. \({UT{\char 94}2}\) ’s human-like qualities allowed it to take second place in BotPrize 2010, a competition to develop human-like bots for Unreal Tournament 2004. This chapter analyzes \({UT{\char 94}2}\) , explains how it achieved its current level of humanness, and discusses insights gained from the competition results that should lead to improved human-like bot performance in future competitions and in videogames in general.

Evolving multi-modal behavior in NPCs

Evolution is often successful in generating complex behaviors, but evolving agents that exhibit distinctly different modes of behavior under different circumstances (multi-modal behavior) is both difficult and time consuming. This paper presents a method for encouraging the evolution of multi-modal behavior in agents controlled by artificial neural networks: A network mutation is introduced that adds enough output nodes to the network to create a new output mode. Each output mode completely defines the behavior of the network, but only one mode is chosen at any one time, based on the output values of preference nodes. With such structure, networks are able to produce appropriate outputs for several modes of behavior simultaneously, and arbitrate between them using preference nodes. This mutation makes it easier to discover interesting multi-modal behaviors in the course of neuroevolution.

Discovering Multimodal Behavior in Ms. Pac-Man through Evolution of Modular Neural Networks.

Ms. Pac-Man is a challenging video game in which multiple modes of behavior are required: Ms. Pac-Man must escape ghosts when they are threats and catch them when they are edible, in addition to eating all pills in each level. Past approaches to learning behavior in Ms. Pac-Man have treated the game as a single task to be learned using monolithic policy representations. In contrast, this paper uses a framework called Modular Multiobjective NEAT (MM-NEAT) to evolve modular neural networks. Each module defines a separate behavior. The modules are used at different times according to a policy that can be human-designed (i.e., multitask) or discovered automatically by evolution. The appropriate number of modules can be fixed or discovered using a genetic operator called module mutation. Several versions of module mutation are evaluated in this paper. Both fixed modular networks and Module Mutation networks outperform monolithic networks and multitask networks. Interestingly, the best networks dedicate modules to critical behaviors (such as escaping when surrounded after luring ghosts near a power pill) that do not follow the customary division of the game into chasing edible and escaping threat ghosts. The results demonstrate that MM-NEAT can discover interesting and effective behavior for agents in challenging games.

Believable Bot Navigation via Playback of Human Traces

Imitation is a powerful and pervasive primitive underlying examples of intelligent behaviour in nature. Can we use it as a tool to help build artificial agents that behave like humans do? This question is studied in the context of the BotPrize competition, a Turing-like test where computer game bots compete by attempting to fool human judges into thinking they are just another human player. One problem faced by such bots is that of human-like navigation within the virtual world. This chapter describes the Human Trace Controller, a component of the \({UT{\char 94}2}\) bot which took second place in the BotPrize 2010 competition. The controller uses a database of recorded human games in order to quickly retrieve and play back relevant segments of human navigation behaviour. Empirical evidence suggests that the method of direct imitation allows the bot to effectively solve several navigation problems while moving in a human-like fashion.

Evolving multimodal behavior through modular multiobjective neuroevolution

Intelligent organisms do not simply perform one task, but exhibit multiple distinct modes of behavior. For instance, humans can swim, climb, write, solve problems, and play sports. To be fully autonomous and robust, it would be advantageous for artificial agents, both in physical and virtual worlds, to exhibit a similar diversity of behaviors. Artificial evolution, in particular neuroevolution [3, 4], is known to be capable of discovering complex agent behavior. This dissertation expands on existing neuroevolution methods, specifically NEAT (Neuro-Evolution of Augmenting Topologies [7]), to make the discovery of multiple modes of behavior possible. More specifically, it proposes four extensions: (1) multiobjective evolution, (2) sensors that are split up according to context, (3) modular neural network structures, and (4) fitness-based shaping. All of these technical contributions are incorporated into the software framework of Modular Multiobjective NEAT (MM-NEAT), which can be downloaded here.

Evolving multimodal networks for multitask games

Intelligent opponent behavior helps make video games interesting to human players. Evolutionary computation can discover such behavior, especially when the game consists of a single task. However, multitask domains, in which separate tasks within the domain each have their own dynamics and objectives, can be challenging for evolution. This paper proposes two methods for meeting this challenge by evolving neural networks: 1) Multitask Learning provides a network with distinct outputs per task, thus evolving a separate policy for each task, and 2) Mode Mutation provides a means to evolve new output modes, as well as a way to select which mode to use at each moment. Multitask Learning assumes agents know which task they are currently facing; if such information is available and accurate, this approach works very well, as demonstrated in the Front/Back Ramming game of this paper. In contrast, Mode Mutation discovers an appropriate task division on its own, which may in some cases be even more powerful than a human-specified task division, as shown in the Predator/Prey game of this paper. These results demonstrate the importance of both Multitask Learning and Mode Mutation for learning intelligent behavior in complex games.

Evolving mario levels in the latent space of a deep convolutional generative adversarial network

Generative Adversarial Networks (GANs) are a machine learning approach capable of generating novel example outputs across a space of provided training examples. Procedural Content Generation (PCG) of levels for video games could benefit from such models, especially for games where there is a pre-existing corpus of levels to emulate. This paper trains a GAN to generate levels for Super Mario Bros using a level from the Video Game Level Corpus. The approach successfully generates a variety of levels similar to one in the original corpus, but is further improved by application of the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). Specifically, various fitness functions are used to discover levels within the latent space of the GAN that maximize desired properties. Simple static properties are optimized, such as a given distribution of tile types. Additionally, the champion A* agent from the 2009 Mario AI competition is used to assess whether a level is playable, and how many jumping actions are required to beat it. These fitness functions allow for the discovery of levels that exist within the space of examples designed by experts, and also guide the search towards levels that fulfill one or more specified objectives.