Stuart W. Card

Information distance based fitness and diversity metrics

Commensurate indicators of diversity and fitness with desirable metric properties are derived from information distances based on Shannon entropy and Kolmogorov complexity. These metrics measure various useful distances: from an information theoretic characterization of the phenotypic behavior of a candidate model in the population to that of an ideal model of the target systems input-output relationship (fitness); from behavior of one candidate model to that of another (total information diversity); from the information about the target provided by one model to that provided by another (target relevant information diversity); from the code of one model to that of another (genotypic representation diversity); etc. Algorithms are cited for calculating the Shannon entropy based metrics from discrete data and estimating analogs thereof from heuristically binned continuous data; references are cited to methods for estimating the Kolmogorov complexity based metric. Not in the paper, but at the workshop, results will be shown of applying these algorithms to several synthetic and real world data sets: the simplest known deterministic chaotic flow; symbolic regression test functions; industrial process monitoring and control variables; and international political leadership data. Ongoing work is outlined.

A network theoretic analysis of evolutionary algorithms

Network theoretic analyses have been shown to be extremely useful in multiple fields and applications. We propose this approach to study the dynamic behavior of evolutionary algorithms, the first such analysis to the best of our knowledge. Evolving populations are represented as dynamic networks, and we show that changes in population characteristics can be recognized at the level of the networks representing successive generations, with implications for possible improvements in the evolutionary algorithm, e.g., in deciding when a population is prematurely converging, and when a reinitialization of the population may be beneficial to reduce computational effort. In this paper, we show that network-theoretic analyses of evolutionary algorithms help in: (i) studying community-level behaviors, and (ii) using graph properties and metrics to analyze evolutionary algorithms.

An Application of Information Theoretic Selection to Evolution of Models with Continuous-valued Inputs

Information theoretic functionals have significant benefits as compared with traditional error based indicators of fitness and diversity. Mutual Information (MI), various normalizations of it, and similarity and distance metrics derived from it, can be used advantageously in all phases of Genetic Programming (GP), starting with input selection. However, these functionals are based on Shannon’s entropy, which is strictly defined only for discrete random variables, so their application to problems involving continuous valued data requires their generalization and development of robust and efficient algorithms for their calculation. This paper outlines such algorithms and illustrates their application to a noisy continuous valued data set synthesized to test GP symbolic regression systems (Korns, 2007). Information theoretic sufficiency outperforms linear correlation in ranking the relevance of available inputs in this data set. Similar results are obtained on inputs filtered by functions that ‘fold’ the data, thereby destroying information; ranking these intermediate evolutionary forms, sufficiency again outperforms correlation. Sufficiency also exhibits a distinct threshold separating irrelevant terms from terms that are indeed relevant in regression of these test problems. As a less computationally costly alternative to rankings of entire populations, tournament selection is often used; on this data set, for pairwise tournament selection, sufficiency greatly outperforms correlation. Multi-objective ranking, considering also information theoretic necessity to prefer appropriately filtered inputs (over corresponding raw inputs with excess entropies), is foreshadowed.

Ensemble selection for evolutionary learning using information theory and price's theorem

This paper presents an information theoretic perspective on design and analysis of evolutionary algorithms. Indicators of solution quality are developed and applied not only to individuals but also to ensembles, thereby ensuring information diversity. Prices Theorem is extended to show how joint indicators can drive reproductive sampling rate of potential parental pairings. Heritability of mutual information is identified as a key issue.

Multiobjective information theoretic ensemble selection

In evolutionary learning, the sine qua non is evolvability, which requires heritability of fitness and a balance between exploitation and exploration. Unfortunately, commonly used fitness measures, such as root mean squared error (RMSE), often fail to reward individuals whose presence in the population is needed to explain important data variance; and indicators of diversity generally are not only incommensurate with those of fitness but also essentially arbitrary. Thus, due to poor scaling, deception, etc., apparently relatively high fitness individuals in early generations may not contain the building blocks needed to evolve optimal solutions in later generations. To reward individuals for their potential incremental contributions to the solution of the overall problem, heritable information theoretic functionals are developed that incorporate diversity considerations into fitness, explicitly identifying building blocks suitable for recombination (e.g. for non-random mating). Algorithms for estimating these functionals from either discrete or continuous data are illustrated by application to input selection in a high dimensional industrial process control data set. Multiobjective information theoretic ensemble selection is shown to avoid some known feature selection pitfalls.

Evolution of Space-Partitioning Forest for Anomaly Detection

Previous work proposed a fast one-class anomaly detector using an ensemble of random half-space partitioning trees. The method was shown to be effective and efficient for detecting anomalies in streaming data. However, the parameters were pre-defined, so the random partitions of the data space might not be optimal. Therefore, the aims of this study were to: (a) give some mathematical analysis of the random partitioning trees; and (b) explore optimizing forests for anomaly detection using evolutionary algorithms.

Rule networks in learning classifier systems

Interrelationships between rules can be used to develop network models that can usefully represent the dynamics of Learning Classifier Systems. We examine two different kinds of rule networks and study their significance by testing them on the 20-mux problem. Through this experimentation, we establish that there is latent information in the evolving rule networks alongside the usual information that we gain from the XCS. We analyze these interrelationships using metrics from Network Science. We also show that these network measures behave as reliable indicators of rule set convergence.