Dominik Fisch

Quantitative Emergence -- A Refined Approach Based on Divergence Measures

The article addresses the phenomenon of emergence from a technical viewpoint. A technical system exhibits emergence when it has certain kinds of properties or qualities that are irreducible in the sense that they are not traceable to the constituent parts of the system. In particular, we show how emergence in technical systems can be detected and measured gradually using techniques from the field of probability theory and information theory. To detect or measure emergence we observe the system and extract characteristic attributes from those observations. As an extension of earlier work in the field, we propose emergence measures that are well-suited for continuous attributes (or hybrid attribute sets) using either non-parametric or model-based probability density estimation techniques. We also replace the known entropy-based emergence measures by divergence measures for probability densities (e.g., the Kullback-Leibler divergence or the Hellinger distance). We discuss advantages and drawbacks of these measures by means of some simulation experiments using artificial data sets and a real-world data set from the field of intrusion detection.

So near and yet so far: New insight into properties of some well-known classifier paradigms

This article provides some new insight into the properties of four well-established classifier paradigms, namely support vector machines (SVM), classifiers based on mixture density models (CMM), fuzzy classifiers (FCL), and radial basis function neural networks (RBF). It will be shown that these classifiers can be formulated in a way such that they are functionally equivalent or at least highly similar. The interpretation of a specific classifier as being an SVM, CMM, FCL, or RBF then only depends on the objective function and the optimization algorithm used to adjust the parameters. The properties of these four paradigms, however, are very different: a discriminative classifier such as an SVM is expected to have optimal generalization capabilities on new data, a generative classifier such as a CMM also aims at modeling the processes from which the observed data originate, and a comprehensible classifier such as an FCL is intended to be parameterized and understood by human domain experts. We will discuss the advantages and disadvantages of these properties and show how they can be measured numerically in order to compare these classifiers. In such a way, the article aims at supporting a practitioner in assessing the properties of classifier paradigms and in selecting or combining certain paradigms for a given application problem.

SwiftRule: Mining Comprehensible Classification Rules for Time Series Analysis

In this article, we provide a new technique for temporal data mining which is based on classification rules that can easily be understood by human domain experts. Basically, time series are decomposed into short segments, and short-term trends of the time series within the segments (e.g., average, slope, and curvature) are described by means of polynomial models. Then, the classifiers assess short sequences of trends in subsequent segments with their rule premises. The conclusions gradually assign an input to a class. As the classifier is a generative model of the processes from which the time series are assumed to originate, anomalies can be detected, too. Segmentation and piecewise polynomial modeling are done extremely fast in only one pass over the time series. Thus, the approach is applicable to problems with harsh timing constraints. We lay the theoretical foundations for this classifier, including a new distance measure for time series and a new technique to construct a dynamic classifier from a static one, and demonstrate its properties by means of various benchmark time series, for example, Lorenz attractor time series, energy consumption in a building, or ECG data.

Learning by teaching versus learning by doing: Knowledge exchange in organic agent systems

“Learning by doing” and “learning by teaching” are two important concepts for human education. In this article, we demonstrate that these learning concepts can also be realized by intelligent, so-called organic computing systems. These organic agents either improve their skills by themselves, eventually assisted by a teacher, or they teach each other by exchanging learned rules. We show that “learning by teaching” may reduce the query costs for teachers and allow for a proactive behavior of organic agents: Before certain situations emerge in their environment, they are already enabled to deal with that situations. We also show that “learning by teaching” may be problematic in cases where different agents are expected to have—at least partially“different skills. Then, incautious knowledge exchange may yield a performance degradation. There are many possible application fields for these organic systems, e.g., distributed intrusion detection, robotics, or sensor networks.

A framework for large-scale simulation of collaborative intrusion detection systems

Distributed intrusion detection and prevention play an increasingly important role in securing computer networks. In a distributed intrusion detection system, information about the current situation and knowledge about attacks are exchanged, aggregated, fused, and correlated in a cooperative manner to overcome the limitations of conventional centralized intrusion detection systems. However, this distributed approach introduces new challenges such as self-organization and efficient communication techniques. In this paper we propose a novel framework for developing, simulating, and deploying a distributed intrusion detection system that consists of several collaborating agents. The framework provides a programming interface and comprises all essential communication and synchronization methods that enables self-organized collaboration in a completely distributed manner. In two experiments we demonstrate the performance and capabilities of our implementation by simulating a large-scale worm outbreak and a one-to-many attack. Furthermore, we present two applications of our framework to show how collaboration of agents can be used to detect one-to-many attacks and how detection performance benefits from cooperation of agents.

Collaborative Learning by Knowledge Exchange

Humans act efficiently in a dynamic environment by learning from each other. Thus, it would be highly desirable to enable intelligent distributed systems, e.g., multi-agent systems, smart sensor networks, or teams of robots, to behave in a way which follows that biological archetype. The constituents of a such a distributed system may learn in a collaborative way by communicating locally learned classification rules, for instance. This article first gives an overview of the techniques that we have developed for knowledge exchange. Then, their application is demonstrated in a realistic scenario, collaborative detection of attacks to a computer network.

Novelty-Aware Attack Recognition – Intrusion Detection with Organic Computing Techniques

A typical task of intrusion detection systems is to detect known kinds of attacks by analyzing network traffic. In this article, we will take a step forward and enable such a system to recognize very new kinds of attacks by means of novelty-awareness mechanisms. That is, an intrusion detection system will be able to recognize deficits in its own knowledge and to react accordingly. It will present a learned rule premise to the system administrator which will then be labeled, i.e., extended by an appropriate conclusion. In this article, we present new techniques for novelty-aware attack recognition based on probabilistic rule modeling techniques and demonstrate how these techniques can successfully be applied to intrusion benchmark data. The proposed novelty-awareness techniques may also be used in other application fields by intelligent technical systems (e.g., organic computing systems) to resolve problems with knowledge deficits in a self-organizing way.

Divergence Measures as a Generalised Approach to Quantitative Emergence

A technical system exhibits emergence when it has certain properties or qualities that can be termed to be irreducible in the sense that they are not traceable down to the constituent parts of the system. The article summarises three techniques for emergence detection and emergence measurement that were proposed by members of the Organic Computing community. These techniques are based on information-theoretic and probabilistic viewpoints: the discrete entropy difference discussed in detail in the previous article, the Hellinger distance which is a divergence measure for probability densities, and an iterative approach motivated by divergence measures. Advantages and drawbacks of these measures are demonstrated by means of some simulation experiments using artificial data sets. It is shown that these techniques are able to deal with different kinds of emergent phenomena such as transitions from chaos to order, concept drift, or novelty. That is, with these techniques it is possible to cover a wide range of possible applications.

Towards automation of knowledge understanding

Seven novel interestingness measures are presented that allow to evaluate different aspects of probabilistic generative classifiers.Three case studies utilizing 21 artificial and real-world benchmark data sets illustrate the usefulness of our measures in different application scenarios.We show that our measures can help researchers in three ways: the training process of a classifier can be improved, the trained classifier can be evaluated and simplified if desired, and during the application phase the classifier can be automatically supervised using interestingness evaluations. After data selection, pre-processing, transformation, and feature extraction, knowledge extraction is not the final step in a data mining process. It is then necessary to understand this knowledge in order to apply it efficiently and effectively. Up to now, there is a lack of appropriate techniques that support this significant step. This is partly due to the fact that the assessment of knowledge is often highly subjective, e.g., regarding aspects such as novelty or usefulness. These aspects depend on the specific knowledge and requirements of the data miner. There are, however, a number of aspects that are objective and for which it is possible to provide appropriate measures. In this article we focus on classification problems and use probabilistic generative classifiers based on mixture density models that are quite common in data mining applications. We define objective measures to assess the informativeness, uniqueness, importance, discrimination, representativity, uncertainty, and distinguishability of rules contained in these classifiers numerically. These measures not only support a data miner in evaluating results of a data mining process based on such classifiers. As we will see in illustrative case studies, they may also be used to improve the data mining process itself or to support the later application of the extracted knowledge.