HuaMing Huang

Rank-based outlier detection

We propose a new approach for outlier detection, based on a ranking measure that focuses on the question of whether a point is ‘central’ for its nearest neighbours. Using our notations, a low cumulative rank implies that the point is central. For instance, a point centrally located in a cluster has a relatively low cumulative sum of ranks because it is among the nearest neighbours of its own nearest neighbours, but a point at the periphery of a cluster has a high cumulative sum of ranks because its nearest neighbours are closer to each other than the point. Use of ranks eliminates the problem of density calculation in the neighbourhood of the point and this improves the performance. Our method performs better than several density-based methods on some synthetic data sets as well as on some real data sets.

Algorithms for detecting outliers via clustering and ranks

Rank-based algorithms provide a promising approach for outlier detection, but currently used rank-based measures of outlier detection suffer from two deficiencies: first they assign a large value to an object near a cluster whose density is high even through the object may not be an outlier and second the distance between the object and its nearest cluster plays a mild role though its rank with respect to its neighbor. To correct for these deficiencies we introduce the concept of modified-rank and propose new algorithms for outlier detection based on this concept. Our method performs better than several density-based methods, on some synthetic data sets as well as on some real data sets.

Clustering-Based Anomaly Detection Approaches

This chapter explores anomaly detection approaches based on explicit identification of clusters in a data set. Points that are not within a cluster become candidates to be considered anomalies. Variations among algorithms result in evaluating the relative anomalousness of points that are near (but not inside) a cluster, and also the points at the periphery of a cluster.

Algorithms for Time Series Data

Many practical problems involve data that arrive over time, and are hence in a strict temporal sequence. As discussed in Chap. 5, treating the data as a set, while ignoring the time-stamp, loses information essential to the problem. Treating the time-stamp as just another dimension (on par with other relevant dimensions such as dollar amounts) can only confuse the matter: the occurrence of other attribute values at a specific time instant can mean something quite different from the same attribute values occurring at another time, depending on the immediately preceding values. Such dependencies necessitate considering time as a special aspect of the data for explicit modeling, and treating the data as a sequence rather than a set. Hence anomaly detection for time-sequenced data requires algorithms that are substantially different from those discussed in the previous chapters.

Model-Based Anomaly Detection Approaches

Many data sets are described by models that may capture the underlying processes that lead to generation of data, describing a presumed functional or relational relationship between relevant variables. Such models permit comprehension and concise description of the data sets, facilitating identification of data points that are not consistent with such a description.

Distance-Based Anomaly Detection Approaches

In this chapter we consider anomaly detection based on distance (similarity) measures. Our approach is to explore various possible scenarios in which an anomaly may arise. To keep things simple, in most of the chapter we illustrate basic concepts using one-dimensional observations. Distance based algorithms, proposed by researchers, are presented in Chap. 6.

Distance and Density Based Approaches

In Chap. 3, we discussed distance based approaches for anomaly detection; however there the focus was to illustrate how distances can be measured and minor perturbation in proposed distance can change the outcome; illustrated by simple examples. In this chapter we consider anomaly detection techniques that depend on the distances and densities. The densities can be global or local to the point of concern.

Rank Based Approaches

Density-based methodology that exploits k-neighborhood of a data point has many good features. For instance, it is independent of the distribution of the data and is capable of detecting isolated objects. However it has some shortcomings:

An online anomalous time series detection algorithm for univariate data streams

We address the online anomalous time series detection problem among a set of series, combining three simple distance measures. This approach, akin to control charts, makes it easy to determine when a series begins to differ from other series. Empirical evidence shows that this novel online anomalous time series detection algorithm performs very well, while being efficient in terms of time complexity, when compared to approaches previously discussed in the literature.