Adam Summerville

Automatic mapping of NES games with mappy

Game maps are useful for human players, general-game-playing agents, and data-driven procedural content generation. These maps are generally made by hand-assembling manually-created screen-shots of game levels. Besides being tedious and error-prone, this approach requires additional effort for each new game and level to be mapped. The results can still be hard for humans or computational systems to make use of, privileging visual appearance over semantic information. We describe a software system, Mappy, that produces a good approximation of a linked map of rooms given a Nintendo Entertainment System game program and a sequence of button inputs exploring its world. In addition to visual maps, Mappy outputs grids of tiles (and how they change over time), positions of non-tile objects, clusters of similar rooms that might in fact be the same room, and a set of links between these rooms. We believe this is a necessary step towards developing larger corpora of high-quality semantically-annotated maps for PCG via machine learning and other applications.

CHARDA: Causal Hybrid Automata Recovery via Dynamic Analysis

We propose and evaluate a new technique for learning hybrid automata automatically by observing the runtime behavior of a dynamical system. Working from a sequence of continuous state values and predicates about the environment, CHARDA recovers the distinct dynamic modes, learns a model for each mode from a given set of templates, and postulates causal guard conditions which trigger transitions between modes. Our main contribution is the use of information-theoretic measures (1)~as a cost function for data segmentation and model selection to penalize over-fitting and (2)~to determine the likely causes of each transition. CHARDA is easily extended with different classes of model templates, fitting methods, or predicates. In our experiments on a complex videogame character, CHARDA successfully discovers a reasonable over-approximation of the characters true behaviors. Our results also compare favorably against recent work in automatically learning probabilistic timed automata in an aircraft domain: CHARDA exactly learns the modes of these simpler automata.

Mechanics automatically recognized via interactive observation: jumping

Jumping has been an important mechanic since its introduction in Donkey Kong. It has taken a variety of forms and shown up in numerous games, with each jump having a different feel. In this paper, we use a modified Nintendo Entertainment System (NES) emulator to semi-automatically run experiments on a large subset (~30%) of NES platform games. We use these experiments to build models of jumps from different developers, series, and games across the history of the console. We then examine these models to gain insights into different forms of jumping and their associated feel.

From Mechanics to Meaning

While generative approaches to game design offer great promise, systems can only reliably generate what they can “understand,” which is often represented in a limited, implicit form in hand-crafted evaluation functions or constructive rules. Proceduralist readings, a semiformal approach for interpreting the meaning of a game based on its underlying processes and interactions in conjunction with aesthetic and cultural cues, offer a novel, systematic approach to game understanding. We formalize proceduralist argumentation as a logic program that performs static reasoning over game specifications to derive higher level meanings, as part of Gemini, a bidirectional game analysis and generation system.

Bad News: An Experiment in Computationally Assisted Performance

Dreams of the prospect of computational narrative suggest a future of deeply interactive and personalized fictional experiences that engage our empathy. But the gulf between our current moment and that future is vast. How do we begin to bridge that divide now, both for learning more specifics of these potentials and to create experiences today that can have some of their impact on audiences? We present Bad News, a combination of theatrical performance practices, computational support, and Wizard-of-Oz interaction techniques. These allow for rich, real-time interaction with a procedurally generated world. We believe our approach could enable other research groups to explore similar territory—and the resulting experience is engaging and affecting in ways that help strengthen the case for our envisioned futures and also makes the case for trying to field such experiences today (e.g., in experimental theater or location-based entertainment contexts). Bad News is a realized game enjoyed by players with varying degrees of performance experience, and won the Innovative Game Design track of the 2016 ACM Conference on Human Factors in Computing Systems (CHI) Student Game Competition.

Translating Player Dialogue into Meaning Representations Using LSTMs

In this paper, we present a novel approach to natural language understanding that utilizes context-free grammars (CFGs) in conjunction with sequence-to-sequence (seq2seq) deep learning. Specifically, we take a CFG authored to generate dialogue for our target application, a videogame, and train a long short-term memory (LSTM) recurrent neural network (RNN) to translate the surface utterances that it produces to traces of the grammatical expansions that yielded them. Critically, we already annotated the symbols in this grammar for the semantic and pragmatic considerations that our game’s dialogue manager operates over, allowing us to use the grammatical trace associated with any surface utterance to infer such information. From preliminary offline evaluation, we show that our RNN translates utterances to grammatical traces (and thereby meaning representations) with great accuracy.

Automated game design learning

While general game playing is an active field of research, the learning of game design has tended to be either a secondary goal of such research or it has been solely the domain of humans. We propose a field of research. Automated Game Design Learning (AGDL), with the direct purpose of learning game designs directly through interaction with games in the mode that most people experience games: via play. We detail existing work that touches the edges of this field, describe current successful projects in AGDL and the theoretical foundations that enable them, point to promising apphcations enabled by AGDL, and discuss next steps for this exciting area of study. The key moves of AGDL are to use game programs as the ultimate source of truth about their own design, and to make these design properties available to other systems and avenues of inquiry.