Jun'ichi Takeuchi

On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms

Outlier detection is a fundamental issue in data mining, specifically in fraud detection, network intrusion detection, network monitoring, etc. SmartSifter is an outlier detection engine addressing this problem from the viewpoint of statistical learning theory. This paper provides a theoretical basis for SmartSifter and empirically demonstrates its effectiveness. SmartSifter detects outliers in an on-line process through the on-line unsupervised learning of a probabilistic model (using a finite mixture model) of the information source. Each time a datum is input SmartSifter employs an on-line discounting learning algorithm to learn the probabilistic model. A score is given to the datum based on the learned model with a high score indicating a high possibility of being a statistical outlier. The novel features of SmartSifter are: (1) it is adaptive to non-stationary sources of data; (2) a score has a clear statistical/information-theoretic meaning; (3) it is computationally inexpensive; and (4) it can handle both categorical and continuous variables. An experimental application to network intrusion detection shows that SmartSifter was able to identify data with high scores that corresponded to attacks, with low computational costs. Further experimental application has identified a number of meaningful rare cases in actual health insurance pathology data from Australias Health Insurance Commission.

A unifying framework for detecting outliers and change points from non-stationary time series data

We are concerned with the issues of outlier detection and change point detection from a data stream. In the area of data mining, there have been increased interest in these issues since the former is related to fraud detection, rare event discovery, etc., while the latter is related to event/trend by change detection, activity monitoring, etc. Specifically, it is important to consider the situation where the data source is non-stationary, since the nature of data source may change over time in real applications. Although in most previous work outlier detection and change point detection have not been related explicitly, this paper presents a unifying framework for dealing with both of them on the basis of the theory of on-line learning of non-stationary time series. In this framework a probabilistic model of the data source is incrementally learned using an on-line discounting learning algorithm, which can track the changing data source adaptively by forgetting the effect of past data gradually. Then the score for any given data is calculated to measure its deviation from the learned model, with a higher score indicating a high possibility of being an outlier. Further change points in a data stream are detected by applying this scoring method into a time series of moving averaged losses for prediction using the learned model. Specifically we develop an efficient algorithms for on-line discounting learning of auto-regression models from time series data, and demonstrate the validity of our framework through simulation and experimental applications to stock market data analysis.

On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms

Outlier detection is a fundamental issue in data mining, specifically in fraud detection, network intrusion detection, network monitoring, etc. SmartSifter is an outlier detection engine addressing this problem from the viewpoint of statistical learning theory. This paper provides a theoretical basis for SmartSifter and empirically demonstrates its effectiveness. SmartSifter detects outliers in an on-line process through the on-line unsupervised learning of a probabilistic model (using a finite mixture model) of the information source. Each time a datum is input SmartSifter employs an on-line discounting learning algorithm to learn the probabilistic model. A score is given to the datum based on the learned model with a high score indicating a high possibility of being a statistical outlier. The novel features of SmartSifter are: (1) it is adaptive to non-stationary sources of data; (2) a score has a clear statistical/information-theoretic meaning; (3) it is computationally inexpensive; and (4) it can handle both categorical and continuous variables. An experimental application to network intrusion detection shows that SmartSifter was able to identify data with high scores that corresponded to attacks, with low computational costs. Further experimental application has identified a number of meaningful rare cases in actual health insurance pathology data from Australias Health Insurance Commission.

A unifying framework for detecting outliers and change points from time series

We are concerned with the issue of detecting outliers and change points from time series. In the area of data mining, there have been increased interest in these issues since outlier detection is related to fraud detection, rare event discovery, etc., while change-point detection is related to event/trend change detection, activity monitoring, etc. Although, in most previous work, outlier detection and change point detection have not been related explicitly, this paper presents a unifying framework for dealing with both of them. In this framework, a probabilistic model of time series is incrementally learned using an online discounting learning algorithm, which can track a drifting data source adaptively by forgetting out-of-date statistics gradually. A score for any given data is calculated in terms of its deviation from the learned model, with a higher score indicating a high possibility of being an outlier. By taking an average of the scores over a window of a fixed length and sliding the window, we may obtain a new time series consisting of moving-averaged scores. Change point detection is then reduced to the issue of detecting outliers in that time series. We compare the performance of our framework with those of conventional methods to demonstrate its validity through simulation and experimental applications to incidents detection in network security.

Discovering outlier filtering rules from unlabeled data: combining a supervised learner with an unsupervised learner

This paper is concerned with the problem of detecting outliers from unlabeled data. In prior work we have developed SmartSifter, which is an on-line outlier detection algorithm based on unsupervised learning from data. On the basis of SmartSifter this paper yields a new framework for outlier filtering using both supervised and unsupervised learning techniques iteratively in order to make the detection process more effective and more understandable. The outline of the framework is as follows: In the first round, for an initial dataset, we run SmartSifter to give each data a score, with a high score indicating a high possibility of being an outlier. Next, giving positive labels to a number of higher scored data and negative labels to a number of lower scored data, we create labeled examples. Then we construct an outlier filtering rule by supervised learning from them. Here the rule is generated based on the principle of minimizing extended stochastic complexity. In the second round, for a new dataset, we filter the data using the constructed rule, then among the filtered data, we run SmartSifter again to evaluate the data in order to update the filtering rule. Applying of our framework to the network intrusion detection, we demonstrate that 1) it can significantly improve the accuracy of SmartSifter, and 2) outlier filtering rules can help the user to discover a general pattern of an outlier group.

Mining traffic data from probe-car system for travel time prediction

We are developing a technique to predict travel time of a vehicle for an objective road section, based on real time traffic data collected through a probe-car system. In the area of Intelligent Transport System (ITS), travel time prediction is an important subject. Probe-car system is an upcoming data collection method, in which a number of vehicles are used as moving sensors to detect actual traffic situation. It can collect data concerning much larger area, compared with traditional fixed detectors. Our prediction technique is based on statistical analysis using AR model with seasonal adjustment and MDL (Minimum Description Length) criterion. Seasonal adjustment is used to handle periodicities of 24 hours in traffic data. Alternatively, we employ state space model, which can handle time series with periodicities. It is important to select really effective data for prediction, among the data from widespread area, which are collected via probe-car system. We do this using MDL criterion. That is, we find the explanatory variables that really have influence on the future travel time. In this paper, we experimentally show effectiveness of our method using probe-car data collected in Nagoya Metropolitan Area in 2002.

Asymptotically minimax regret by Bayes mixtures

We study the problem of data compression, gambling and prediction of a sequence x/sup n/ = x/sub 1/x/sub 2/...x/sub n/ from a certain alphabet X, in terms of regret (Shtarkov 1988) and redundancy with respect to a general exponential family, a general smooth family, and also Markov sources. In particular, we show that variants of Jeffreys mixture asymptotically achieve their minimax values.

Polynomial learnability of probabilistic concepts with respect to the Kullback-Leibler divergence

We consider the problem of efficient learning of probabilistic concepts (p-concepts), in the sense defined by Kearns and Schapire [KS90] and by Yamanishi [Yam90]. Their models extend the PAC-learning model of Valiant [Val84] to the learning scenario in which the target concept or function is stochastic rather than deterministic as in Valiants original model. In this paper, we consider the learnability for p-concepts with respect to the classic ‘Kullback-Leibler divergence’ (KL divergence) as the distance measure between p-concepts. First, we show that the notion of polynomial time learnability of p-concepts using the KL divergence is in fact equivalent to the same notion using any of the distances considered in [KS90] and [Yam90], the quadratic, variation, and Hellinger distances. As a corollary, it follows that a wide range of classes of p-concepts which were shown to be polynomially learnable with respect to the quadratic distance in [KS90] are also learnable with respect to the KL divergence. The sample and time complexity of algorithms that would be obtained by the above general equivalence, however, are far from optimal. We present a polynomial learning algorithm with a reasonable sample complexity for the important class of convex linear combinations of p-concepts. We also develop simple and versatile techniques for obtaining sample complexity bounds for learning classes of p-concepts with respect to the KL-divergence and quadratic distance, and apply them to produce bounds for the classes of probabilistic finite state acceptors (automata), probabilistic decision lists, and linear combinations.

α-parallel prior and its properties

It is known that the Jeffreys prior plays an important role in statistical inference. In this paper, we generalize the Jeffreys prior from the point of view of information geometry and introduce a one-parameter family of prior distributions, which we named the /spl alpha/-parallel priors. The /spl alpha/-parallel prior is defined as the parallel volume element with respect to the /spl alpha/-connection and coincides with the Jeffreys prior when /spl alpha/=0. Further, we analyze asymptotic behavior of the various estimators such as the projected Bayes estimator (the estimator obtained by projecting the Bayes predictive density onto the original class of distributions) and the minimum description length (MDL) estimator, when the /spl alpha/-parallel prior is used. The difference of these estimators from maximum-likelihood estimator (MLE) due to the /spl alpha/-prior is shown to be regulated by an invariant vector field of the statistical model. Although the Jeffreys prior always exists, the existence of /spl alpha/-parallel prior with /spl alpha/ /spl ne/ 0 is not always guaranteed. Hence, we consider conditions for the existence of the /spl alpha/-parallel prior, elucidating the conjugate symmetry in a statistical model.

An incident analysis system NICTER and its analysis engines based on data mining techniques

Malwares are spread all over cyberspace and often lead to serious security incidents. To grasp the present trends of malware activities, there are a number of ongoing network monitoring projects that collect large amount of data such as network traffic and IDS logs. These data need to be analyzed in depth since they potentially contain critical symptoms, such as an outbreak of new malware, a stealthy activity of botnet and a new type of attack on unknown vulnerability, etc. We have been developing the Network Incident analysis Center for Tactical Emergency Response (NICTER), which monitors a wide range of networks in real-time. The NICTER deploys several analysis engines taking advantage of data mining techniques in order to analyze the monitored traffics. This paper describes a brief overview of the NICTER, and its data mining based analysis engines, such as Change Point Detector (CPD), Self-Organizing Map analyzer (SOM analyzer) and Incident Forecast engine (IF).