Ivo Gonçalves

Random sampling technique for overfitting control in genetic programming

One of the areas of Genetic Programming (GP) that, in comparison to other Machine Learning methods, has seen fewer research efforts is that of generalization. Generalization is the ability of a solution to perform well on unseen cases. It is one of the most important goals of any Machine Learning method, although in GP only recently has this issue started to receive more attention. In this work we perform a comparative analysis of a particularly interesting configuration of the Random Sampling Technique (RST) against the Standard GP approach. Experiments are conducted on three multidimensional symbolic regression real world datasets, the first two on the pharmacokinetics domain and the third one on the forestry domain. The results show that the RST decreases overfitting on all datasets. This technique also improves testing fitness on two of the three datasets. Furthermore, it does so while producing considerably smaller and less complex solutions. We discuss the possible reasons for the good performance of the RST, as well as its possible limitations.

On the Generalization Ability of Geometric Semantic Genetic Programming

Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming (GP) that searches directly the space of the underlying semantics of the programs. The fitness landscape seen by the GSGP variation operators is unimodal with a linear slope by construction and, consequently, easy to search. Despite this advantage, the offspring produced by these operators grow very quickly. A new implementation of the same operators was proposed that computes the semantics of the offspring without having to explicitly build their syntax. This allowed GSGP to be used for the first time in real-life multidimensional datasets. GSGP presented a surprisingly good generalization ability, which was justified by some properties of the geometric semantic operators. In this paper, we show that the good generalization ability of GSGP was the result of a small implementation deviation from the original formulation of the mutation operator, and that without it the generalization results would be significantly worse. We explain the reason for this difference, and then we propose two variants of the geometric semantic mutation that deterministically and optimally adapt the mutation step. They reveal to be more efficient in learning the training data, and they also achieve a competitive generalization in only a single operator application. This provides a competitive alternative when performing semantic search, particularly since they produce small individuals and compute fast.

Deep learning networks for off-line handwritten signature recognition

Reliable identification and verification of off-line handwritten signatures from images is a difficult problem with many practical applications. This task is a difficult vision problem within the field of biometrics because a signature may change depending on psychological factors of the individual. Motivated by advances in brain science which describe how objects are represented in the visual cortex, advanced research on deep neural networks has been shown to work reliably on large image data sets. In this paper, we present a deep learning model for off-line handwritten signature recognition which is able to extract high-level representations. We also propose a two-step hybrid model for signature identification and verification improving the misclassification rate in the well-known GPDS database.

Arbitrarily Close Alignments in the Error Space: A Geometric Semantic Genetic Programming Approach

This paper shows how arbitrarily close alignments in the error space can be achieved by Genetic Programming. The consequences for the generalization ability of the resulting individuals are explored.

Prediction of forest aboveground biomass: an exercise on avoiding overfitting

Mapping and understanding the spatial distribution of forest aboveground biomass (AGB) is an important and challenging task. This paper describes an exercise of predicting the forest AGB of Guinea-Bissau, West Africa, using synthetic aperture radar data and measurements of tree size collected in field campaigns. Several methods were attempted, from linear regression to different variants and techniques of Genetic Programming (GP), including the cutting edge geometric semantic GP approach. The results were compared between each other in terms of root mean square error and correlation between predicted and expected values of AGB. None of the methods was able to produce a model that generalizes well to unseen data or significantly outperforms the model obtained by the state-of-the-art methodology, and the latter was also not better than a simple linear model. We conclude that the AGB prediction is a difficult problem, aggravated by the small size of the available data set.

Semantic Learning Machine: A Feedforward Neural Network Construction Algorithm Inspired by Geometric Semantic Genetic Programming

Geometric Semantic Genetic Programming (GSGP) is a recently proposed form of Genetic Programming in which the fitness landscape seen by its variation operators is unimodal with a linear slope by construction and, consequently, easy to search. This is valid across all supervised learning problems. In this paper we propose a feedforward Neural Network construction algorithm derived from GSGP. This algorithm shares the same fitness landscape as GSGP, which allows an efficient search to be performed on the space of feedforward Neural Networks, without the need to use backpropagation. Experiments are conducted on real-life multidimensional symbolic regression datasets and results show that the proposed algorithm is able to surpass GSGP, with statistical significance, in terms of learning the training data. In terms of generalization, results are similar to GSGP.

Unsure when to stop?: ask your semantic neighbors

In iterative supervised learning algorithms it is common to reach a point in the search where no further induction seems to be possible with the available data. If the search is continued beyond this point, the risk of overfitting increases significantly. Following the recent developments in inductive semantic stochastic methods, this paper studies the feasibility of using information gathered from the semantic neighborhood to decide when to stop the search. Two semantic stopping criteria are proposed and experimentally assessed in Geometric Semantic Genetic Programming (GSGP) and in the Semantic Learning Machine (SLM) algorithm (the equivalent algorithm for neural networks). The experiments are performed on real-world high-dimensional regression datasets. The results show that the proposed semantic stopping criteria are able to detect stopping points that result in a competitive generalization for both GSGP and SLM. This approach also yields computationally efficient algorithms as it allows the evolution of neural networks in less than 3 seconds on average, and of GP trees in at most 10 seconds. The usage of the proposed semantic stopping criteria in conjunction with the computation of optimal mutation/learning steps also results in small trees and neural networks.

An evolutionary system for ozone concentration forecasting

Nowadays, with more than 50 % of the world’s population living in urban areas, cities are facing important environmental challenges. Among them, air pollution has emerged as one of the most important concerns, taking into account the social costs related to the effect of polluted air. According to a report of the World Health Organization, approximately seven million people die each year from the effects of air pollution. Despite this fact, the same report suggests that cities could greatly improve their air quality through local measures by exploiting modern and efficient solutions for smart infrastructures. Ideally, this approach requires insights of how pollutant levels change over time in specific locations. To tackle this problem, we present an evolutionary system for the prediction of pollutants levels based on a recently proposed variant of genetic programming. This system is designed to predict the amount of ozone level, based on the concentration of other pollutants collected by sensors disposed in critical areas of a city. An analysis of data related to the region of Yuen Long (one of the most polluted areas of China), shows the suitability of the proposed system for addressing the problem at hand. In particular, the system is able to predict the ozone level with greater accuracy with respect to other techniques that are commonly used to tackle similar forecasting problems.

Pruning Techniques for Mixed Ensembles of Genetic Programming Models

The objective of this paper is to define an effective strategy for building an ensemble of Genetic Programming (GP) models. Ensemble methods are widely used in machine learning due to their features: they average out biases, they reduce the variance and they usually generalize better than single models. Despite these advantages, building ensemble of GP models is not a well-developed topic in the evolutionary computation community. To fill this gap, we propose a strategy that blends individuals produced by standard syntax-based GP and individuals produced by geometric semantic genetic programming, one of the newest semantics-based method developed in GP. In fact, recent literature showed that combining syntax and semantics could improve the generalization ability of a GP model. Additionally, to improve the diversity of the GP models used to build up the ensemble, we propose different pruning criteria that are based on correlation and entropy, a commonly used measure in information theory. Experimental results,obtained over different complex problems, suggest that the pruning criteria based on correlation and entropy could be effective in improving the generalization ability of the ensemble model and in reducing the computational burden required to build it.

Towards the development of a complete GP system on an FPGA using geometric semantic operators

Genetic Programming (GP) has been around for over two decades and has been used in a wide range of practical applications producing human competitive results in several domains. In this paper we present a discussion and a proposal of a GP algorithm that could be conveniently implemented on an embedded system, as part of a broader research project that pursues the implementation of a complete GP system in a Field Programmable Gate Array (FPGA). Motivated by the significant time savings associated with such a platform, as well as low power consumption, low maintenance requirements, small size of the system and the possibility of performing several parallel processes. The proposal is focused on the Geometric Semantic Genetic Programming (GSGP) approach that has been recently introduced with promising results. GSGP induces a unimodal fitness landscape, simplifying the search process. The experimental work considers five variants of GSGP, that incorporate local search strategies, optimal mutations and alignment in error space. Best results were obtained by a simple variant that uses both the optimal mutation step and the standard geometric semantic mutation, using three difficult real-world problems to evaluate the methods, outperforming the original GSGP formulation in terms of fitness and empirical convergence.