Bernhard Sick

Evolutionary optimization of radial basis function classifiers for data mining applications

In many data mining applications that address classification problems, feature and model selection are considered as key tasks. That is, appropriate input features of the classifier must be selected from a given (and often large) set of possible features and structure parameters of the classifier must be adapted with respect to these features and a given data set. This paper describes an evolutionary algorithm (EA) that performs feature and model selection simultaneously for radial basis function (RBF) classifiers. In order to reduce the optimization effort, various techniques are integrated that accelerate and improve the EA significantly: hybrid training of RBF networks, lazy evaluation, consideration of soft constraints by means of penalty terms, and temperature-based adaptive control of the EA. The feasibility and the benefits of the approach are demonstrated by means of four data mining problems: intrusion detection in computer networks, biometric signature verification, customer acquisition with direct marketing methods, and optimization of chemical production processes. It is shown that, compared to earlier EA-based RBF optimization techniques, the runtime is reduced by up to 99% while error rates are lowered by up to 86%, depending on the application. The algorithm is independent of specific applications so that many ideas and solutions can be transferred to other classifier paradigms.

Online Signature Verification With Support Vector Machines Based on LCSS Kernel Functions

In this paper, a new technique for online signature verification or identification is proposed. The technique integrates a longest common subsequences (LCSS) detection algorithm which measures the similarity of signature time series into a kernel function for support vector machines (SVM). LCSS offers the possibility to consider the local variability of signals such as the time series of pen-tip coordinates on a graphic tablet, forces on a pen, or inclination angles of a pen measured during a signing process. Consequently, the similarity of two signature time series can be determined in a more reliable way than with other measures. A proprietary database with signatures of 153 test persons and the SVC 2004 benchmark database are used to show the properties of the new SVM-LCSS. We investigate its parameterization and compare it to SVM with other kernel functions such as dynamic time warping (DTW). Our experiments show that SVM with the LCSS kernel authenticate persons very reliably and with a performance which is significantly better than that of the best comparing technique, SVM with DTW kernel.

Online Segmentation of Time Series Based on Polynomial Least-Squares Approximations

The paper presents SwiftSeg, a novel technique for online time series segmentation and piecewise polynomial representation. The segmentation approach is based on a least-squares approximation of time series in sliding and/or growing time windows utilizing a basis of orthogonal polynomials. This allows the definition of fast update steps for the approximating polynomial, where the computational effort depends only on the degree of the approximating polynomial and not on the length of the time window. The coefficients of the orthogonal expansion of the approximating polynomial-obtained by means of the update steps-can be interpreted as optimal (in the least-squares sense) estimators for average, slope, curvature, change of curvature, etc., of the signal in the time window considered. These coefficients, as well as the approximation error, may be used in a very intuitive way to define segmentation criteria. The properties of SwiftSeg are evaluated by means of some artificial and real benchmark time series. It is compared to three different offline and online techniques to assess its accuracy and runtime. It is shown that SwiftSeg-which is suitable for many data streaming applications-offers high accuracy at very low computational costs.

Online Intrusion Alert Aggregation with Generative Data Stream Modeling

Alert aggregation is an important subtask of intrusion detection. The goal is to identify and to cluster different alerts-produced by low-level intrusion detection systems, firewalls, etc.-belonging to a specific attack instance which has been initiated by an attacker at a certain point in time. Thus, meta-alerts can be generated for the clusters that contain all the relevant information whereas the amount of data (i.e., alerts) can be reduced substantially. Meta-alerts may then be the basis for reporting to security experts or for communication within a distributed intrusion detection system. We propose a novel technique for online alert aggregation which is based on a dynamic, probabilistic model of the current attack situation. Basically, it can be regarded as a data stream version of a maximum likelihood approach for the estimation of the model parameters. With three benchmark data sets, we demonstrate that it is possible to achieve reduction rates of up to 99.96 percent while the number of missing meta-alerts is extremely low. In addition, meta-alerts are generated with a delay of typically only a few seconds after observing the first alert belonging to a new attack instance.

On-line motif detection in time series with SwiftMotif

This article presents SwiftMotif, a novel technique for on-line motif detection in time series. With this technique, frequently occurring temporal patterns or anomalies can be discovered, for instance. The motif detection is based on a fusion of methods from two worlds: probabilistic modeling and similarity measurement techniques are combined with extremely fast polynomial least-squares approximation techniques. A time series is segmented with a data stream segmentation method, the segments are modeled by means of normal distributions with time-dependent means and constant variances, and these models are compared using a divergence measure for probability densities. Then, using suitable clustering algorithms based on these similarity measures, motifs may be defined. The fast time series segmentation and modeling techniques then allow for an on-line detection of previously defined motifs in new time series with very low run-times. SwiftMotif is suitable for real-time applications, accounts for the uncertainty associated with the occurrence of certain motifs, e.g., due to noise, and considers local variability (i.e., uniform scaling) in the time domain. This article focuses on the mathematical foundations and the demonstration of properties of SwiftMotif-in particular accuracy and run-time-using some artificial and real benchmark time series.

Feature selection for intrusion detection: an evolutionary wrapper approach

With the ongoing growth of the Internet, intrusion detection systems (IDS) play an increasing role in securing communication networks. Particularly, where sensitive and confidential information is stored or transmitted, there is a vital importance of security. In the past few years, soft-computing techniques (especially neural networks) found their way more and more into the research area of intrusion detection and are now an inherent part of it. Although, feature selection is an important task for almost all neural network applications, only very few investigations dealing with any type of automated feature selection are known in the area of intrusion detection. This article sets out an evolutionary algorithm (EA) that performs the tasks of feature selection and architecture optimization for radial basis function (RBF) networks automatically. With the feature selection process proposed, it is possible to reduce the number of input features significantly, which is very important due to the fact that the neural networks can effectively be prevented from overfitting. Furthermore, reducing the number of input features also reduces the time needed for feature extraction as well as the execution time for the trained RBF networks and allows the extraction of well interpretable rules. The results (depending on the attack type, about 2 up to 5 features are needed for attack detection on an average) are demonstrated for seven attack types using the DARPA 1998 intrusion detection evaluation data.

Evolutionary optimization of radial basis function networks for intrusion detection

Feature selection and architecture optimization are two key tasks in most neural network applications. Appropriate input features must be selected from a given (and often large) set of possible features and architecture parameters of the network such as the number of hidden neurons or learning parameters must be adapted with respect to the selected features and a learning data set. This article sets out an evolutionary algorithm (EA) that performs the tasks simultaneously for radial basis function (RBF) networks. The feasibility and the benefits of this approach are demonstrated in an application in the area of computer security: the detection of attacks (intrusive behavior) in computer networks. The EA, however, is independent from the application example given so that the ideas and solutions may easily be transferred to other applications and even other neural network paradigms. In the application example investigated overall classification rates of about 99.4% (average of eight attack types) can be reached for independent validation data.

Quantitative Emergence -- A Refined Approach Based on Divergence Measures

The article addresses the phenomenon of emergence from a technical viewpoint. A technical system exhibits emergence when it has certain kinds of properties or qualities that are irreducible in the sense that they are not traceable to the constituent parts of the system. In particular, we show how emergence in technical systems can be detected and measured gradually using techniques from the field of probability theory and information theory. To detect or measure emergence we observe the system and extract characteristic attributes from those observations. As an extension of earlier work in the field, we propose emergence measures that are well-suited for continuous attributes (or hybrid attribute sets) using either non-parametric or model-based probability density estimation techniques. We also replace the known entropy-based emergence measures by divergence measures for probability densities (e.g., the Kullback-Leibler divergence or the Hellinger distance). We discuss advantages and drawbacks of these measures by means of some simulation experiments using artificial data sets and a real-world data set from the field of intrusion detection.

Emergence in organic computing systems: discussion of a controversial concept

Philosophy of mind has investigated the emergent behavior of complex systems for more than a century. However, terms such as “weak” or “strong” emergence are hardly applicable to intelligent technical systems. Organic Computing has the goal to utilize concepts such as emergence and self-organization to build complex technical systems. At first glance this seems to be a contradiction, but: These systems must be reliable and trustworthy! In order to measure, to control, and even to design emergence, a new notion or definition of emergence is needed. This article first describes the definition of emergence as used in philosophy of mind because this definition is often misunderstood or misinterpreted. Then, some very recent approaches for definitions of emergence in more or less technical contexts are discussed from the viewpoint of Organic Computing. The article concludes with some new thoughts that may help to come to a unifying notion of emergence in intelligent technical systems.

So near and yet so far: New insight into properties of some well-known classifier paradigms

This article provides some new insight into the properties of four well-established classifier paradigms, namely support vector machines (SVM), classifiers based on mixture density models (CMM), fuzzy classifiers (FCL), and radial basis function neural networks (RBF). It will be shown that these classifiers can be formulated in a way such that they are functionally equivalent or at least highly similar. The interpretation of a specific classifier as being an SVM, CMM, FCL, or RBF then only depends on the objective function and the optimization algorithm used to adjust the parameters. The properties of these four paradigms, however, are very different: a discriminative classifier such as an SVM is expected to have optimal generalization capabilities on new data, a generative classifier such as a CMM also aims at modeling the processes from which the observed data originate, and a comprehensible classifier such as an FCL is intended to be parameterized and understood by human domain experts. We will discuss the advantages and disadvantages of these properties and show how they can be measured numerically in order to compare these classifiers. In such a way, the article aims at supporting a practitioner in assessing the properties of classifier paradigms and in selecting or combining certain paradigms for a given application problem.