Berend Weel

Online Gait Learning for Modular Robots with Arbitrary Shapes and Sizes

This paper addresses a principal problem of in vivo evolution of modular multi-cellular robots. To evolve robot morphologies and controllers in real-space and real-time we need a generic learning mechanism that enables arbitrary modular shapes to obtain a suitable gait quickly after ‘birth’. In this study we investigate a reinforcement learning method and conduct simulation experiments using robot morphologies with different size and complexity. The experiments give insights into the online dynamics of gait learning, the distribution of lucky / unlucky runs and their dependence on the size and complexity of the modular robotic organisms.

MONEE: using parental investment to combine open-ended and task-driven evolution

This paper is inspired by a vision of self-sufficient robot collectives that adapt autonomously to deal with their environment and to perform user-defined tasks at the same time. We introduce the monee algorithm as a method of combining open-ended (to deal with the environment) and task-driven (to satisfy user demands) adaptation of robot controllers through evolution. A number of experiments with simulated e-pucks serve as proof of concept and show that with monee, the robots adapt to cope with the environment and to perform multiple tasks. Our experiments indicate that monee distributes the tasks evenly over the robot collective without undue emphasis on easy tasks.

Right on the MONEE: combining task- and environment-driven evolution

Evolution can be employed for two goals. Firstly, to provide a force for adaptation to the environment as it does in nature and in many artificial life implementations - this allows the evolving population to survive. Secondly, evolution can provide a force for optimisation as is mostly seen in evolutionary robotics research - this causes the robots to do something useful. We propose the MONEE algorithmic framework as an approach to combine these two facets of evolution: to combine environment-driven and task-driven evolution. To achieve this, MONEE employs environment-driven and task-based parent selection schemes in parallel. We test this approach in a simulated experimental setting where the robots are tasked to collect two different kinds of puck. MONEE allows the robots to adapt their behaviour to successfully tackle these tasks while ensuring an equitable task distribution at no cost in task performance through a market-based mechanism. In environments that discourage robots performing multiple tasks and in environments where one task is easier than the other, MONEEs market mechanism prevents the population completely focussing on one task.

Hyperneat versus rl power for online gait learning in modular robots

This paper addresses a principal problem of in vivo evolution of modular multi-cellular robots, where robot ‘babies’ can be produced with arbitrary shapes and sizes. In such a system we need a generic learning mechanism that enables newborn morphologies to obtain a suitable gait quickly after ‘birth’. In this study we investigate and compare the reinforcement learning method RL PoWeR with HyperNEAT. We conduct simulation experiments using robot morphologies with different size and complexity. The experiments give insights into the differences in solution quality and algorithm efficiency, suggesting that reinforcement learning is the preferred option for this online learning problem.

Testing diversity-enhancing migration policies for hybrid on-line evolution of robot controllers

We investigate on-line on-board evolution of robot controllers based on the so-called hybrid approach (island-based). Inherently to this approach each robot hosts a population (island) of evolving controllers and exchanges controllers with other robots at certain times. We compare different exchange (migration) policies in order to optimize this evolutionary system and compare the best hybrid setup with the encapsulated and distributed alternatives. We conclude that adding a difference-based migrant selection scheme increases the performance.

On-line Evolution of Controllers for Aggregating Swarm Robots in Changing Environments

Optimization of an engineering system or component makes a series of changes in the initial random solution(s) iteratively to form the final optimal shape. When multiple conflicting objectives are considered, recent studies on innovization revealed the fact that the set of Pareto-optimal solutions portray certain common design principles. In this paper, we consider a 14-variable bi-objective design optimization of a MEMS device and identify a number of such common design principles through a recently proposed automated innovization procedure. Although these design principles are found to exist among near-Paretooptimal solutions, the main crux of this paper lies in a demonstration of temporal evolution of these principles during the course of optimization. The results reveal that certain important design principles start to evolve early on, whereas some detailed design principles get constructed later during optimization. Interestingly, there exists a simile between evolution of design principles with that of human evolution. Such information about the hierarchy of key design principles should enable designers to have a deeper understanding of their problems.

Online gait learning for modular robots with arbitrary shapes and sizes

Evolutionary robotics using real hardware is currently restricted to evolving robot controllers, but the technology for evolvable morphologies is advancing quickly. Rapid prototyping (3D printing) and automated assembly are the main enablers of robotic systems where robot offspring can be produced based on a blueprint that specifies the morphologies and the controllers of the parents. This article addresses the problem of gait learning in newborn robots whose morphology is unknown in advance. We investigate a reinforcement learning method and conduct simulation experiments using robot morphologies with different size and complexity. We establish that reinforcement learning does the job well and that it outperforms two alternative algorithms. The experiments also give insights into the online dynamics of gait learning and into the influence of the size, shape, and morphological complexity of the modular robots. These insights can potentially be used to predict the viability of modular robotic organisms before they are constructed.

Comparing generic parameter controllers for EAs

Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist and, additionally, no comparisons or in depth analyses of these generic controllers are available in literature. This paper presents an extensive comparison of such generic parameter control methods, including a number of novel controllers based on reinforcement learning which are introduced here. We conducted experiments with different EAs and test problems in an one-off setting, i.e. relatively long runs with controllers used out-of-the-box with no tailoring to the problem at hand. Results reveal several interesting insights regarding the effectiveness of parameter control, the niche applications/EAs, the effect of continuous treatment of parameters and the influence of noise and randomness on control.