Giorgos Karafotias

Parameter Control in Evolutionary Algorithms: Trends and Challenges

More than a decade after the first extensive overview on parameter control, we revisit the field and present a survey of the state-of-the-art. We briefly summarize the development of the field and discuss existing work related to each major parameter or component of an evolutionary algorithm. Based on this overview, we observe trends in the area, identify some (methodological) shortcomings, and give recommendations for future research.

On-line evolution of robot controllers by an encapsulated evolution strategy

This paper describes and experimentally evaluates the viability of the (µ +1) ON-LINE evolutionary algorithm for on-line adaptation of robot controllers. Secondly, it explores the parameter space for this algorithm and identifies four important parameters: the population size µ, the re-evaluation rate ρ, the mutation step-size σ and the controller evaluation period τ. Subsequently, it investigates their influence on controller performance, stability of behaviour and speed of adaptation. The results indicate that the encapsulated on-line evolutionary approach is a viable one and merits further research. In agreement with existing research, the mutation step-size σ proves to be of overriding importance to finding good solutions. Specific to on-line evolution, the results show that longer evaluation times greatly benefit the quality of controllers as well as stability of behaviour and speed of adaptation.

An algorithm for distributed on-line, on-board evolutionary robotics

This paper presents part of an endeavour towards robots and robot collectives that can adapt their controllers autonomously and self-sufficiently and so independently learn to cope with situations unforeseen by their designers. We introduce the Embodied Distributed Evolutionary Algorithm (DEA) for on-line, on-board adaptation of robot controllers. We experimentally evaluate DEA using a number of well-known tasks in the evolutionary robotics field to determine whether it is a viable implementation of on-line, on-board evolution. We compare it to the encapsulated mu + 1 ON- LINE algorithm in terms of (the stability of) task performance and the sensitivity to parameter settings. Experiments show that DEA provides an effective method for on-line, on-board adaptation of robot controllers. Compared to mu + 1 ON- LINE, in terms of performance there is no clear winner, but in terms of sensitivity to parameter settings and stability of performance DEA is significantly better than mu + 1 ON- LINE.

Generic parameter control with reinforcement learning

Parameter control in Evolutionary Computing stands for an approach to parameter setting that changes the parameters of an Evolutionary Algorithm (EA) on-the-fly during the run. In this paper we address the issue of a generic and parameter-independent controller that can be readily plugged into an existing EA and offer performance improvements by varying the EA parameters during the problem solution process. Our approach is based on a careful study of Reinforcement Learning (RL) theory and the use of existing RL techniques. We present experiments using various state-of-the-art EAs solving different difficult problems. Results show that our RL control method has very good potential in improving the quality of the solution found without requiring additional resources or time and with minimal effort from the designer of the application.

A generic approach to parameter control

On-line control of EA parameters is an approach to parameter setting that offers the advantage of values changing during the run. In this paper, we investigate parameter control from a generic and parameter-independent perspective. We propose a generic control mechanism that is targeted to repetitive applications, can be applied to any numeric parameter and is tailored to specific types of problems through an off-line calibration process. We present proof-of-concept experiments using this mechanism to control the mutation step size of an Evolutionary Strategy (ES). Results show that our method is viable and performs very well, compared to the tuning approach and traditional control methods.

Evaluating Reward Definitions for Parameter Control

Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist. Recently, successful parameter control methods based on Reinforcement Learning (RL) have been suggested for one-off applications, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. However, the reward function used was not investigated in depth, though it is a non-trivial factor with an important impact on the performance of a RL mechanism. In this paper, we address this issue by defining and comparing four alternative reward functions for such generic and RL-based EA parameter controllers. We conducted experiments with different EAs, test problems and controllers and results showed that the simplest reward function performs at least as well as the others, making it an ideal choice for generic out-of-the-box parameter control.

It's fate: a self-organising evolutionary algorithm

We introduce a novel evolutionary algorithm where the centralized oracle ---the selection-reproduction loop--- is replaced by a distributed system of Fate Agents that autonomously perform the evolutionary operations. This results in a distributed, situated, and self-organizing EA, where candidate solutions and Fate Agents co-exist and co-evolve. Our motivation comes from evolutionary swarm robotics where candidate solutions evolve in real time and space. As a first proof-of-concept, however, here we test the algorithm with abstract function optimization problems. The results show that the Fate Agents EA is capable of evolving good solutions and it can cope with noise and changing fitness landscapes. Furthermore, an analysis of algorithm behavior also shows that this EA successfully regulates population sizes and adapts its parameters.

Parameter control: strategy or luck?

Parameter control mechanisms in evolutionary algorithms (EAs) dynamically change the values of the EA parameters during a run. Research over the last two decades has delivered ample examples where an EA using a parameter control mechanism outperforms its static version with fixed parameter values. However, very few have investigated why such parameter control approaches perform better. In principle, it could be the case that using different parameter values alone is already sufficient and EA performance can be improved without sophisticated control strategies. This paper investigates whether very simple random variation in parameter values during an evolutionary run can already provide improvements over static values.

Comparing generic parameter controllers for EAs

Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist and, additionally, no comparisons or in depth analyses of these generic controllers are available in literature. This paper presents an extensive comparison of such generic parameter control methods, including a number of novel controllers based on reinforcement learning which are introduced here. We conducted experiments with different EAs and test problems in an one-off setting, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. Results reveal several interesting insights regarding the effectiveness of parameter control, the niche applications/EAs, the effect of continuous treatment of parameters and the influence of noise and randomness on control.

Fate agent evolutionary algorithms with self-adaptive mutation

Fate Agent EAs form a novel flavour or subclass in EC. The idea is to decompose the main loop of traditional evolutionary algorithms into three independently acting forces, implemented by the so-called Fate Agents, and create an evolutionary process by injecting these agents into a population of candidate solutions. This paper introduces an extension to the original concept, adding a mechanism to self-adapt the mutation of the Breeder Agents. The method improves the behaviour of the original Fate Agent EA on dynamically changing fitness landscapes.