Justin K. Pugh

Quality Diversity: A New Frontier for Evolutionary Computation

While evolutionary computation and evolutionary robotics take inspiration from nature, they have long focused mainly on problems of performance optimization. Yet evolution in nature can be interpreted as more nuanced than a process of simple optimization. In particular, natural evolution is a divergent search that optimizes locally within each niche as it simultaneously diversifies. This tendency to discover both quality and diversity at the same time differs from many of the conventional algorithms of machine learning, and also thereby suggests a different foundation for inferring the approach of greatest potential for evolutionary algorithms. In fact, several recent evolutionary algorithms called quality diversity (QD) algorithms(e.g. novelty search with local competition and MAP-Elites) have drawn inspiration from this more nuanced view, aiming to fill a space of possibilities with the best possible example of each type of achievable behavior. The result is a new class of algorithms that return an archive of diverse, high-quality behaviors in a single run. The aim in this paper is to study the application of QD algorithms in challenging environments (in particular complex mazes) to establish their best practices for ambitious domains in the future. In addition to providing insight into cases when QD succeeds and fails, a new approach is investigated that hybridizes multiple views of behaviors (called behavior characterizations) in the same run, which succeeds in overcoming some of the challenges associated with searching for QD with respect to a behavior characterization that is not necessarily sufficient for generating both quality and diversity at the same time.

Confronting the Challenge of Quality Diversity

In contrast to the conventional role of evolution in evolutionary computation (EC) as an optimization algorithm, a new class of evolutionary algorithms has emerged in recent years that instead aim to accumulate as diverse a collection of discoveries as possible, yet where each variant in the collection is as fit as it can be. Often applied in both neuroevolution and morphological evolution, these new quality diversity (QD) algorithms are particularly well-suited to evolutions inherent strengths, thereby offering a promising niche for EC within the broader field of machine learning. However, because QD algorithms are so new, until now no comprehensive study has yet attempted to systematically elucidate their relative strengths and weaknesses under different conditions. Taking a first step in this direction, this paper introduces a new benchmark domain designed specifically to compare and contrast QD algorithms. It then shows how the degree of alignment between the measure of quality and the behavior characterization (which is an essential component of all QD algorithms to date) impacts the ultimate performance of different such algorithms. The hope is that this initial study will help to stimulate interest in QD and begin to unify the disparate ideas in the area.

Evolving multimodal controllers with HyperNEAT

Natural brains effectively integrate multiple sensory modalities and act upon the world through multiple effector types. As researchers strive to evolve more sophisticated neural controllers, confronting the challenge of multimodality is becoming increasingly important. As a solution, this paper presents a principled new approach to exploiting indirect encoding to incorporate multimodality based on the HyperNEAT generative neuroevolution algorithm called the multi-spatial substrate (MSS). The main idea is to place each input and output modality on its own independent plane. That way, the spatial separation of such groupings provides HyperNEAT an a priori hint on which neurons are associated with which that can be exploited from the start of evolution. To validate this approach, the MSS is compared with more conventional approaches to HyperNEAT substrate design in a multiagent domain featuring three input and two output modalities. The new approach both significantly outperforms conventional approaches and reduces the creative burden on the user to design the layout of the substrate, thereby opening formerly prohibitive multimodal problems to neuroevolution.

Searching for Quality Diversity When Diversity is Unaligned with Quality

Inspired by natural evolution’s affinity for discovering a wide variety of successful organisms, a new evolutionary search paradigm has emerged wherein the goal is not to find the single best solution but rather to collect a diversity of unique phenotypes where each variant is as good as it can be. These quality diversity (QD) algorithms therefore must explore multiple promising niches simultaneously. A QD algorithm’s diversity component, formalized by specifying a behavior characterization (BC), not only generates diversity but also promotes quality by helping to overcome deception in the fitness landscape. However, some BCs (particularly those that are unaligned with the notion of quality) do not adequately mitigate deception, rendering QD algorithms unable to discover the best-performing solutions on difficult problems. This paper introduces a solution that enables QD algorithms to pursue arbitrary notions of diversity without compromising their ability to solve hard problems: driving search with multiple BCs simultaneously.

Voxelbuild: a minecraft-inspired domain for experiments in evolutionary creativity

The fields of artificial life and evolutionary robotics have seen growing interest in evolution as a source of creativity, as opposed to a tool for optimization. New intentionally divergent algorithms such as novelty search with local competition (NSLC) and MAP-Elites accordingly attempt to harness evolutions aptitude for divergence in a new search paradigm called quality diversity (QD), which aims to find a wide variety of possible solutions spread across a behavior space. To date, QD has mainly been studied in domains where potential diversity is limited. In anticipation of future, more open-ended applications of QD algorithms, this paper introduces a new domain inspired by the popular Minecraft video game featuring a larger behavior space that is substantially more difficult to exhaust. Preliminary results are presented, showcasing sample block structures built by evolved neural network controllers.

Major evolutionary transitions in the Voxelbuild virtual sandbox game.

Developing a comprehensive theory of open-ended evolution (OEE) depends critically on understanding the mechanisms underlying the major evolutionary transitions; such periods of rapid innovation, s...

An Extended Study of Quality Diversity Algorithms

In a departure from conventional optimization where the goal is to find the best possible solution, a new class of evolutionary algorithms instead search for quality diversity (QD) -- a maximally diverse collection of individuals in which each member is as high-performing as possible. In QD, diversity of behaviors or phenotypes is defined by a behavior characterization (BC) that is typically unaligned with (i.e. orthogonal to) the notion of quality. As experiments in a difficult maze task reinforce, QD algorithms driven by such an unaligned BC are unable to discover the best solutions on sufficiently deceptive problems. This study comprehensively surveys known QD algorithms and introduces several novel variants thereof, including a method for successfully confronting deceptive QD landscapes: driving search with multiple BCs simultaneously.