Danilo Vasconcellos Vargas

Self organizing classifiers and niched fitness

Learning classifier systems are adaptive learning systems which have been widely applied in a multitude of application domains. However, there are still some generalization problems unsolved. The hurdle is that fitness and niching pressures are difficult to balance. Here, a new algorithm called Self Organizing Classifiers is proposed which faces this problem from a different perspective. Instead of balancing the pressures, both pressures are separated and no balance is necessary. In fact, the proposed algorithm possesses a dynamical population structure that self-organizes itself to better project the input space into a map. The niched fitness concept is defined along with its dynamical population structure, both are indispensable for the understanding of the proposed method. Promising results are shown on two continuous multi-step problems. One of which is yet more challenging than previous problems of this class in the literature.

Self organizing classifiers: first steps in structured evolutionary machine learning

Learning classifier systems (LCSs) are evolutionary machine learning algorithms, flexible enough to be applied to reinforcement, supervised and unsupervised learning problems with good performance. Recently, self organizing classifiers were proposed which are similar to LCSs but have the advantage that in its structured population no balance between niching and fitness pressure is necessary. However, more tests and analysis are required to verify its benefits. Here, a variation of the first algorithm is proposed which uses a parameterless self organizing map (SOM). This algorithm is applied in challenging problems such as big, noisy as well as dynamically changing continuous input-action mazes (growing and compressing mazes are included) with good performance. Moreover, a genetic operator is proposed which utilizes the topological information of the SOM’s population structure, improving the results. Thus, the first steps in structured evolutionary machine learning are shown, nonetheless, the problems faced are more difficult than the state-of-art continuous input-action multi-step ones.

Multi-objective phylogenetic algorithm: solving multi-objective decomposable deceptive problems

In general, Multi-objective Evolutionary Algorithms do not guarantee find solutions in the Pareto-optimal set. We propose a new approach for solving decomposable deceptive multi-objective problems that can find all solutions of the Pareto-optimal set. Basically, the proposed approach starts by decomposing the problem into subproblems and, then, combining the found solutions. The resultant approach is a Multi-objective Estimation of Distribution Algorithm for solving relatively complex multi-objective decomposable problems, using a probabilistic model based on a phylogenetic tree. The results show that, for the tested problem, the algorithm can efficiently find all the solutions of the Pareto-optimal set, with better scaling than the hierarchical Bayesian Optimization Algorithm and other algorithms of the state of art.

Novelty-Organizing Team of Classifiers in noisy and dynamic environments

In the real world, the environment is constantly changing with the input variables under the effect of noise. However, few algorithms were shown to be able to work under those circumstances. Here, Novelty-Organizing Team of Classifiers (NOTC) is applied to the continuous action mountain car as well as two variations of it: a noisy mountain car and an unstable weather mountain car. These problems take respectively noise and change of problem dynamics into account. Moreover, NOTC is compared with NeuroEvolution of Augmenting Topologies (NEAT) in these problems, revealing a trade-off between the approaches. While NOTC achieves the best performance in all of the problems, NEAT needs less trials to converge. It is demonstrated that NOTC achieves better performance because of its division of the input space (creating easier problems). Unfortunately, this division of input space also requires a bit of time to bootstrap.

General subpopulation framework and taming the conflict inside populations

Structured evolutionary algorithms have been investigated for some time. However, they have been under explored especially in the field of multi-objective optimization. Despite good results, the use of complex dynamics and structures keep the understanding and adoption rate of structured evolutionary algorithms low. Here, we propose a general subpopulation framework that has the capability of integrating optimization algorithms without restrictions as well as aiding the design of structured algorithms. The proposed framework is capable of generalizing most of the structured evolutionary algorithms, such as cellular algorithms, island models, spatial predator-prey, and restricted mating based algorithms. Moreover, we propose two algorithms based on the general subpopulation framework, demonstrating that with the simple addition of a number of single-objective differential evolution algorithms for each objective, the results improve greatly, even when the combined algorithms behave poorly when evaluated alone at the tests. Most importantly, the comparison between the subpopulation algorithms and their related panmictic algorithms suggests that the competition between different strategies inside one population can have deleterious consequences for an algorithm and reveals a strong benefit of using the subpopulation framework.

Novelty-organizing team of classifiers - A team-individual multi-objective approach to reinforcement learning

In reinforcement learning, there are basically two spaces to search: value-function space and policy space. Consequently, there are two fitness functions each with their associated trade-offs. However, the problem is still perceived as a single-objective one. Here a multi-objective reinforcement learning algorithm is proposed with a structured novelty map population evolving feedforward neural models. It outperforms a gradient based continuous input-output state-of-art algorithm in two problems. Contrary to the gradient based algorithm, the proposed one solves both problems with the same parameters and smaller variance of results. Moreover, the results are comparable even with other discrete action algorithms of the literature as well as neuroevolution methods such as NEAT. The proposed method brings also the novelty map population concept, i.e., a novelty map-based population which is less sensitive to the input distribution and therefore more suitable to create the state space. In fact, the novelty map framework is shown to be less dynamic and more resource efficient than variants of the self-organizing map.

Novelty-organizing classifiers applied to classification and reinforcement learning: towards flexible algorithms

It is widely known that reinforcement learning is a more general problem than supervised learning. In fact, supervised learning can be seen as a class of reinforcement learning problems. However, only a couple of papers tested reinforcement learning algorithms in supervised learning problems. Here we propose a new and simpler way to abstract supervised learning for any reinforcement learning algorithm. Moreover, a new algorithm called Novelty-Organizing Classifiers is developed based on a Novelty Map population that focuses more on the novelty of the inputs than their frequency. A comparison of the proposed method with Self-Organizing Classifiers and BioHel on some datasets is presented. Even though BioHel is specialized in solving supervised learning problems, the results showed only a trade-off between the algorithms. Lastly, results on a maze problem validate the flexibility of the proposed algorithm beyond supervised learning problems. Thus, Novelty-Organizing Classifiers is capable of solving many supervised learning problems as well as a maze problem without changing any parameter at all. Considering the fact that no adaptation of parameters was executed, the proposed algorithms basis seems interestingly flexible.

Evasion Attacks Against Statistical Code Obfuscation Detectors

In the domain of information security, code obfuscation is a feature often employed for malicious purposes. For example there have been quite a few papers reporting that obfuscated JavaScript frequently comes with malicious functionality such as redirecting to external malicious websites. In order to capture such obfuscation, a class of detectors based on statistical features of code, mostly n-grams have been proposed and been claimed to achieve high detection accuracy. In this paper, we formalize a common scenario between defenders who maintain the statistical obfuscation detectors and adversaries who want to evade the detection. Accordingly, we create two kinds of evasion attack methods and evaluate the robustness of statistical detectors under such attacks. Experimental results show that statistical obfuscation detectors can be easily fooled by a sophisticated adversary even in worst case scenarios.

Curious: Searching for Unknown Regions of Space with a Subpopulation-based Algorithm

Intrinsic motivation and novelty search are promising approaches to deal with plateaus, deceptive functions and other exploration problems where using only the main objective function is insufficient. However, it is not clear until now how and if intrinsic motivation (novelty search) can improve single objective algorithms in general. The hurdle is that using multi-objective algorithms to deal with single-objective problems adds an unnecessary overhead such as the search for non-dominated solutions. Here, we propose the Curious algorithm which is the first multi-objective algorithm focused on solving single-objective problems. Curious uses two subpopulations algorithms. One subpopulation is dedicated for improving objective function values and another one is added to search for unknown regions of space based on objective prediction errors. By using a differential evolution operator, genes from individuals in all subpopulations are mixed. In this way, the promising regions (solutions with high fitness) and unknown regions (solutions with high prediction error) are searched simultaneously. Because of thus realized strong yet well controlled novelty search, the algorithm possesses powerful exploration ability and outperforms usual single population based algorithms such as differential evolution. Thus, it demonstrates that the addition of intrinsic motivation is promising and should improve further single objective algorithms in general.

The Relationship Between (Un)Fractured Problems and Division of Input Space

Problems can be categorized as fractured or unfractured ones. A different set of characteristics are needed for learning algorithms to solve each of these two types of problems. However, the exact characteristics needed to solve each type are unclear. This article shows that the division of the input space is one of these characteristics. In other words, a study is presented showing that while fractured problems benefit from a finer division of the input space, unfractured problems benefit from a coarser division of input space. Many open questions still remains. And the article discusses two conjectures which can be used to solve fractured problems more easily.