Bernhard Sick

Transductive active learning – A new semi-supervised learning approach based on iteratively refined generative models to capture structure in data

Abstract Pool-based active learning is a paradigm where users (e.g., domains experts) are iteratively asked to label initially unlabeled data, e.g., to train a classifier from these data. An appropriate selection strategy has to choose unlabeled data for such user queries in an efficient and effective way (in principle, high classification performance at low labeling costs). In our transductive active learning approach we provide a completely labeled data pool (samples are either labeled by the experts or in a semi-supervised way) in each active learning cycle. Thereby, a key aspect is to explore and exploit information about structure in data. Structure in data can be detected and modeled by means of clustering algorithms or probabilistic, generative modeling techniques, for instance. Usually, this is done at the beginning of the active learning process when the data are still unlabeled. In our approach we show how a probabilistic generative model, initially parametrized with unlabeled data, can iteratively be refined and improved when during the active learning process more and more labels became available. In each cycle of the active learning process we use this generative model to label all samples not labeled by an expert so far in order to train the kind of classifier we want to train with the active learning process. Thus, this transductive learning process can be combined with any selection strategy and any kind of classifier. Here, we combine it with the 4DS selection strategy and the CMM probabilistic classifier described in previous work. For 20 publicly available benchmark data sets, we show that this new transductive learning process helps to improve pool-based active learning noticeably.

The responsibility weighted Mahalanobis kernel for semi-supervised training of support vector machines for classification

The responsibility weighted Mahalanobis (RWM) kernel considers structure information in data with help of a parametric density model.It is perfectly suited for semi-supervised learning as the parameters of the density model can be found in an unsupervised way.For semi-supervised learning the RWM kernel outperforms some other kernel functions including the Laplacian kernel (Laplacian SVM). SVM with RWM kernels can be parameterized as easily as an SVM with standard RBF kernels, as known heuristics for the RBF kernel can be transferred to the new kernel.Standard training techniques such as SMO and standard implementations of SVM such as LIBSVM can be used with the RWM kernel without any algorithmic adjustments or extensions.Results are shown for 20 publicly available benchmark data sets. Kernel functions in support vector machines (SVM) are needed to assess the similarity of input samples in order to classify these samples, for instance. Besides standard kernels such as Gaussian (i.e., radial basis function, RBF) or polynomial kernels, there are also specific kernels tailored to consider structure in the data for similarity assessment. In this paper, we will capture structure in data by means of probabilistic mixture density models, for example Gaussian mixtures in the case of real-valued input spaces. From the distance measures that are inherently contained in these models, e.g., Mahalanobis distances in the case of Gaussian mixtures, we derive a new kernel, the responsibility weighted Mahalanobis (RWM) kernel. Basically, this kernel emphasizes the influence of model components from which any two samples that are compared are assumed to originate (that is, the responsible model components). We will see that this kernel outperforms the RBF kernel and other kernels capturing structure in data (such as the LAP kernel in Laplacian SVM) in many applications where partially labeled data are available, i.e., for semi-supervised training of SVM. Other key advantages are that the RWM kernel can easily be used with standard SVM implementations and training algorithms such as sequential minimal optimization, and heuristics known for the parametrization of RBF kernels in a C-SVM can easily be transferred to this new kernel. Properties of the RWM kernel are demonstrated with 20 benchmark data sets and an increasing percentage of labeled samples in the training data.

Let us know your decision: Pool-based active training of a generative classifier with the selection strategy 4DS

In this article, we introduce and investigate 4DS, a new selection strategy for pool-based active training of a generative classifier, namely CMM (classifier based on a probabilistic mixture model). Such a generative classifier aims at modeling the processes underlying the generation of the data. 4DS considers the distance of samples (observations) to the decision boundary, the density in regions, where samples are selected, the diversity of samples in the query set that are chosen for labeling, and, indirectly, the unknown class distribution of the samples by utilizing the responsibilities of the model components for these samples. The combination of the four measures in 4DS is self-optimizing in the sense that the weights of the distance, density, and class distribution measures depend on the currently estimated performance of the classifier. With 17 benchmark data sets it is shown that 4DS outperforms a random selection strategy (baseline method), a pure closest sampling approach, ITDS (information theoretic diversity sampling), DWUS (density-weighted uncertainty sampling), DUAL (dual strategy for active learning), PBAC (prototype based active learning), and 3DS (a technique we proposed earlier that does not consider responsibility information) regarding various evaluation criteria such as ranked performance based on classification accuracy, number of labeled samples (data utilization), and learning speed assessed by the area under the learning curve. It is also shown that-due to the use of responsibility information-4DS solves a key problem of active learning: The class distribution of the samples chosen for labeling actually approximates the unknown true class distribution of the overall data set quite well. With this article, we also pave the way for advanced selection strategies for an active training of discriminative classifiers such as support vector machines or decision trees: We show that responsibility information derived from generative models can successfully be employed to improve the training of those classifiers.

Techniques for knowledge acquisition in dynamically changing environments

Intelligent agents often have the same or similar tasks and sometimes they cooperate to solve a given problem. These agents typically know how to observe their local environment and how to react on certain observations, for instance, and this knowledge may be represented in form of rules. However, many environments are dynamic in the sense that from time to time novel rules are required or old rules become obsolete. In this article we propose and investigate new techniques for knowledge acquisition by novelty detection and reaction as well as obsoleteness detection and reaction that an agent may use for self-adaptation to new situations. For that purpose we consider classifiers based on probabilistic rules. Premises of new rules are learned autonomously while conclusions are either obtained from human experts or from other agents which have learned appropriate rules in the past. By means of knowledge exchange, agents will efficiently be enabled to cope with situations they were not confronted with before. This kind of collaborative intelligence follows the human archetype: Humans are able to learn from each other by communicating learned rules. We demonstrate some properties of the knowledge acquisition techniques using artificial data as well as data from the field of intrusion detection.

On general purpose time series similarity measures and their use as kernel functions in support vector machines

The article addresses the problem of temporal data mining, in particular classification, with support vector machines (SVM). If no application-specific knowledge about the nature of the time series is available, general purpose time series similarity measures can be used as kernel functions in SVM. The article compares several possible similarity measures, namely the linear Euclidean, triangle, polynomial probabilistic (with two variants), and shape space distances (SSD), as well as the nonlinear measures dynamic time warping (DTW), longest common subsequences, and time warp edit distance (TWED). Nonlinear (i.e., elastic) measures take a nonlinear scaling of the time series in the time domain into account. First, these measures are used in combination with a nearest neighbor classifier, then the various similarity measures are taken to compute the kernel matrices for SVM. Simulation experiments with twenty publicly available benchmark data sets show, that with regard to classification accuracy, TWED performs very well over all measures, while SSD is the best linear measure. SSD has the lowest run-times, the fastest nonlinear measure is DTW. These claims are further investigated by applying statistical tests. With the results presented in this article and results from related investigations that are considered as well, we want to support practitioners or scholars in answering the following question: Which measure should be looked at first if accuracy is the most important criterion, if an application is time-critical, or if a compromise is needed?

Knowledge Fusion for Probabilistic Generative Classifiers with Data Mining Applications

If knowledge such as classification rules are extracted from sample data in a distributed way, it may be necessary to combine or fuse these rules. In a conventional approach this would typically be done either by combining the classifiers outputs (e.g., in form of a classifier ensemble) or by combining the sets of classification rules (e.g., by weighting them individually). In this paper, we introduce a new way of fusing classifiers at the level of parameters of classification rules. This technique is based on the use of probabilistic generative classifiers using multinomial distributions for categorical input dimensions and multivariate normal distributions for the continuous ones. That means, we have distributions such as Dirichlet or normal-Wishart distributions over parameters of the classifier. We refer to these distributions as hyperdistributions or second-order distributions. We show that fusing two (or more) classifiers can be done by multiplying the hyperdistributions of the parameters and derive simple formulas for that task. Properties of this new approach are demonstrated with a few experiments. The main advantage of this fusion approach is that the hyperdistributions are retained throughout the fusion process. Thus, the fused components may, for example, be used in subsequent training steps (online training).

Learning from others: Exchange of classification rules in intelligent distributed systems

Learning by an exchange of knowledge and experiences enables humans to act efficiently in a very dynamic environment. Thus, it would be highly desirable to enable intelligent distributed systems to behave in a way which follows that biological archetype. We believe that knowledge exchange will become increasingly important in many application areas such as intrusion detection, driver assistance, or robotics. Constituents of a distributed system such as software agents, cars equipped with smart sensors, or intelligent robots may learn from each other by exchanging knowledge in form of classification rules, for instance. This article proposes techniques for the exchange of classification rules that represent uncertain knowledge. For that purpose, we introduce methods for knowledge acquisition in dynamic environments, for gathering and using meta-knowledge about rules (i.e., experience), and for rule exchange in distributed systems. The methods are based on a probabilistic knowledge modeling approach. We describe the results of two case studies where we show that knowledge exchange (exchange of learned rules) may be superior to information exchange (exchange of raw observations, i.e. samples) and demonstrate that the use of experiences (meta-knowledge concerning the rules) may improve that rule exchange process further. Some possible real application scenarios are sketched briefly and an application in the field of intrusion detection in computer networks is elaborated in more detail.

Deep Learning for solar power forecasting — An approach using AutoEncoder and LSTM Neural Networks

Power forecasting of renewable energy power plants is a very active research field, as reliable information about the future power generation allow for a safe operation of the power grid and helps to minimize the operational costs of these energy sources. Deep Learning algorithms have shown to be very powerful in forecasting tasks, such as economic time series or speech recognition. Up to now, Deep Learning algorithms have only been applied sparsely for forecasting renewable energy power plants. By using different Deep Learning and Artificial Neural Network algorithms, such as Deep Belief Networks, AutoEncoder, and LSTM, we introduce these powerful algorithms in the field of renewable energy power forecasting. In our experiments, we used combinations of these algorithms to show their forecast strength compared to a standard MLP and a physical forecasting model in the forecasting the energy output of 21 solar power plants. Our results using Deep Learning algorithms show a superior forecasting performance compared to Artificial Neural Networks as well as other reference models such as physical models.

Capacity of Low-Voltage Grids for Distributed Generation: Classification by Means of Stochastic Simulations

Without appropriate counteraction, the high amount of installed distributed generators (DG) at the low-voltage distribution level may cause overloading of electrical equipments and violation of voltage limits in many grids. Because of the historically grown low-voltage grids and their local and geographic dependencies, complex grid structures can be found. Thus, the discrimination of grids concerning their DG capacity is a difficult task. We propose a novel three-step classification strategy to distinguish various kinds of low-voltage grids regarding their DG capacity. Our method is based on a stochastic simulation procedure and a subsequent parametric stochastic modeling, which allows for a probability based classification approach. The classification results can be regarded as probabilistic class memberships or, if sharp memberships are required, the class with the maximum probability can be selected. The proposed approach will not only help distribution system operators to face the challenges in future grid planning and focus their work on further enhancement of weak grid structures, but it will also be valuable in choosing relevant grids for detailed surveys. To demonstrate that our approach actually leads to meaningful classification results for real low-voltage grids, we empirically evaluate the results for 300 real rural and suburban grids by comparing them to classification assessments of experts from distribution grid planning practice.

Analysis on Termination of Pedestrians' Gait at Urban Intersections

This paper analyzes pedestrian gait termination at a public urban intersection. Head-tracks of 183 uninstructed, randomly selected pedestrians are acquired by a wide angle stereo camera setup. Analyses of characteristic velocity patterns, deceleration and step parameters are conducted separately for children, adults and elderly people. Results show that stopping patterns at intersections comprise fast stopping within 2 - 3 steps, patterns of slightly slower stopping and increasing deceleration, and, most common, early intended stopping performed over a deceleration time of about 3 s, i.e. over about 6 steps, on average. Step analysis shows that stopping is a combination of step length decrease and step duration increase, especially for the last step.We measured significant lower steady state velocity and step length of the elderly, longer minimum deceleration times for the majorities of the children and of the elderly, and we observed that stopping behavior during the last 2 - 3 steps is very similar for all three classes.