Ezequiel Di Mario

Analysis of Fitness Noise in Particle Swarm Optimization: From Robotic Learning to Benchmark Functions

Population-based learning techniques have been proven to be effective in dealing with noise and are thus promising tools for the optimization of robotic controllers, which have inherently noisy performance evaluations. This article discusses how the results and guidelines derived from tests on benchmark functions can be extended to the fitness distributions encountered in robotic learning. We show that the large-amplitude noise found in robotic evaluations is disruptive to the initial phases of the learning process of PSO. Under these conditions, neither increasing the population size nor increasing the number of iterations are efficient strategies to improve the performance of the learning. We also show that PSO is more sensitive to good spurious evaluations of bad solutions than bad evaluations of good solutions, i.e., there is a non-symmetric effect of noise on the performance of the learning.

A trajectory-based calibration method for stochastic motion models

In this paper, we present a quantitative, trajectory-based method for calibrating stochastic motion models of water-floating robots. Our calibration method is based on the Correlated Random Walk (CRW) model, and consists in minimizing the Kolmogorov-Smirnov (KS) distance between the step length and step angle distributions of real and simulated trajectories generated by the robots. First, we validate this method by calibrating a physics-based motion model of a single 3-cm-sized robot floating at a water/air interface under fluidic agitation. Second, we extend the focus of our work to multi-robot systems by performing a sensitivity analysis of our stochastic motion model in the context of Self-Assembly (SA). In particular, we compare in simulation the effect of perturbing the calibrated parameters on the predicted distributions of self-assembled structures. More generally, we show that the SA of water-floating robots is very sensitive to even small variations of the underlying physical parameters, thus requiring real-time tracking of its dynamics.

A comparison of PSO and Reinforcement Learning for multi-robot obstacle avoidance

The design of high-performing robotic controllers constitutes an example of expensive optimization in uncertain environments due to the often large parameter space and noisy performance metrics. There are several evaluative techniques that can be employed for on-line controller design. Adequate benchmarks help in the choice of the right algorithm in terms of final performance and evaluation time. In this paper, we use multi-robot obstacle avoidance as a benchmark to compare two different evaluative learning techniques: Particle Swarm Optimization and Q-learning. For Q-learning, we implement two different approaches: one with discrete states and discrete actions, and another one with discrete actions but a continuous state space. We show that continuous PSO has the highest fitness overall, and Q-learning with continuous states performs significantly better than Q-learning with discrete states. We also show that in the single robot case, PSO and Q-learning with discrete states require a similar amount of total learning time to converge, while the time required with Q-learning with continuous states is significantly larger. In the multi-robot case, both Q-learning approaches require a similar amount of time as in the single robot case, but the time required by PSO can be significantly reduced due to the distributed nature of the algorithm.

A distributed noise-resistant Particle Swarm Optimization algorithm for high-dimensional multi-robot learning

Population-based learning techniques have been proven to be effective in dealing with noise in numerical benchmark functions and are thus promising tools for the high-dimensional optimization of controllers for multiple robots with limited sensing capabilities, which have inherently noisy performance evaluations. In this article, we apply a statistical technique called Optimal Computing Budget Allocation to improve the performance of Particle Swarm Optimization in the presence of noise for a multi-robot obstacle avoidance benchmark task. We present a new distributed PSO OCBA algorithm suitable for resource-constrained mobile robots due to its low requirements in terms of memory and limited local communication. Our results from simulation show that PSO OCBA outperforms other techniques for dealing with noise, achieving a more consistent progress and a better estimate of the ground-truth performance of candidate solutions. We then validate our simulations with real robot experiments where we compare the controller learned with our proposed algorithm to a potential field controller for obstacle avoidance in a cluttered environment. We show that they both achieve a high performance through different avoidance behaviors.

Distributed Particle Swarm Optimization for Limited Time Adaptation in Autonomous Robots

Evaluative techniques offer a tremendous potential for on-line controller design. However, when the optimization space is large and the performance metric is noisy, the time needed to properly evaluate candidate solutions becomes prohibitively large and, as a consequence, the overall adaptation process becomes extremely time consuming. Distributing the adaptation process reduces the required time and increases robustness to failure of individual agents. In this paper, we analyze the role of the four algorithmic parameters that determine the total evaluation time in a distributed implementation of a Particle Swarm Optimization algorithm. For a multi-robot obstacle avoidance case study, we explore in simulation the lower boundaries of these parameters with the goal of reducing the total evaluation time so that it is feasible to implement the adaptation process within a limited amount of time determined by the robots’ energy autonomy. We show that each parameter has a different impact on the final fitness and propose some guidelines for choosing these parameters for real robot implementations.

Distributed Learning of Cooperative Robotic Behaviors using Particle Swarm Optimization

In this paper we study the automatic synthesis of robotic controllers for the coordinated movement of multiple mobile robots. The algorithm used to learn the controllers is a noise-resistant version of Particle Swarm Optimization, which is applied in two different settings: centralized and distributed learning. In centralized learning, every robot runs the same controller and the performance is evaluated with a global metric. In the distributed learning, robots run different controllers and the performance is evaluated independently on each robot with a local metric. Our results from learning in simulation show that it is possible to learn a cooperative task in a fully distributed way employing a local metric, and we validate the simulations with real robot experiments where the best solutions from distributed and centralized learning achieve similar performances.

The Effect of the Environment in the Synthesis of Robotic Controllers: A Case Study in Multi-Robot Obstacle Avoidance using Distributed Particle Swarm Optimization

The ability to move in complex environments is a fundamental requirement for robots to be a part of our daily lives. While in simple environments it is usually straightforward for human designers to foresee the different conditions a robot will be exposed to, for more complex environments the human design of high-performing controllers becomes a challenging task, especially when the on-board resources of the robots are limited. In this article, we use a distributed implementation of Particle Swarm Optimization to design robotic controllers that are able to navigate around obstacles of different shape and size. We analyze how the behavior and performance of the controllers differ based on the environment where learning takes place, showing that different arenas lead to different avoidance behaviors. We also test the best controllers in environments not encountered during learning, both in simulation and with real robots, and show that no single learning environment is able to generate a behavior general and robust enough to succeed in all testing environments.

Distributed Particle Swarm Optimization using Optimal Computing Budget Allocation for multi-robot learning

Particle Swarm Optimization (PSO) is a population-based metaheuristic that can be applied to optimize controllers for multiple robots using only local information. In order to cope with noise in the robotic performance evaluations, different reevaluation strategies were proposed in the past. In this article, we apply a statistical technique called Optimal Computing Budget Allocation to improve the performance of distributed PSO in the presence of noise. In particular, we compare a distributed PSO OCBA algorithm suitable for resource-constrained mobile robots with a centralized version that uses global information for the allocation. We show that the distributed PSO OCBA outperforms a previous distributed noise-resistant PSO variant, and that the performance of the distributed PSO OCBA approaches that of the centralized one as the communication radius is increased. We also explore different parametrizations of the PSO OCBA algorithm, and show that the choice of parameter values differs from previous guidelines proposed for stand-alone OCBA.

SwarmViz: An open-source visualization tool for Particle Swarm Optimization

Particle Swarm Optimization (PSO) is a metaheuristic for solving high dimensional optimization problems. Due to the large number of dimensions usually employed with PSO, it is not trivial to visualize and monitor the progress of the algorithm. Because of this, adjusting the parameters that govern the dynamics of the swarm for a specific problem becomes challenging. In this article, we present SwarmViz, an open-source visualization tool for PSO. Through SwarmViz, users are able to set up PSO experiments on canonical benchmark functions or input data from external experiments (e.g., learning robotic controllers), and to visualize the optimization process with state-of-the-art visualization tools. SwarmViz has two main goals. First, to enable researchers to monitor the progress of their specific optimization problem and adjust the relevant PSO parameters. Second, to give a visual insight about PSO to students in the scope of teaching optimization techniques. We demonstrate the features of the software through examples on well-known numerical benchmark functions and a case study on the optimization of a robotic controller.

The Role of Environmental and Controller Complexity in the Distributed Optimization of Multi-Robot Obstacle Avoidance

The ability to move in complex environments is a fundamental requirement for robots to be a part of our daily lives. Increasing the controller complexity may be a desirable choice in order to obtain an improved performance. However, these two aspects may pose a considerable challenge on the optimization of robotic controllers. In this paper, we study the trade-offs between the complexity of reactive controllers and the complexity of the environment in the optimization of multi-robot obstacle avoidance for resource-constrained platforms. The optimization is carried out in simulation using a distributed, noise-resistant implementation of Particle Swarm Optimization, and the resulting controllers are evaluated both in simulation and with real robots. We show that in a simple environment, linear controllers with only two parameters perform similarly to more complex non-linear controllers with up to twenty parameters, even though the latter ones require more evaluation time to be learned. In a more complicated environment, we show that there is an increase in performance when the controllers can differentiate between front and backwards sensors, but increasing further the number of sensors and adding non-linear activation functions provide no further benefit. In both environments, augmenting reactive control laws with simple memory capabilities causes the highest increase in performance. We also show that in the complex environment the performance measurements are noisier, the optimal parameter region is smaller, and more iterations are required for the optimization process to converge.