Paul F. Evangelista

Taming the Curse of Dimensionality in Kernels and Novelty Detection

The curse of dimensionality is a well known but not entirely well-understood phenomena. Too much data, in terms of the number of input variables, is not always a good thing. This is especially true when the problem involves unsupervised learning or supervised learning with unbalanced data (many negative observations but minimal positive observations). This paper addresses two issues involving high dimensional data: The first issue explores the behavior of kernels in high dimensional data. It is shown that variance, especially when contributed by meaningless noisy variables, confounds learning methods. The second part of this paper illustrates methods to overcome dimensionality problems with unsupervised learning utilizing subspace models. The modeling approach involves novelty detection with the one-class SVM.

Some properties of the Gaussian kernel for one class learning

This paper proposes a novel approach for directly tuning the gaussian kernel matrix for one class learning. The popular gaussian kernel includes a free parameter, σ, that requires tuning typically performed through validation. The value of this parameter impacts model performance significantly. This paper explores an automated method for tuning this kernel based upon a hill climbing optimization of statistics obtained from the kernel matrix.

Data Fusion for Outlier Detection through Pseudo-ROC Curves and Rank Distributions

This paper proposes a novel method of fusing models for classification of unbalanced data. The unbalanced data contains a majority of healthy (negative) instances, and a minority of unhealthy (positive) instances. The applicability of this type of classification problem with security applications inspired the naming of such problems as security classification problems (SCP). The area under the ROC curve (AUC) is the metric utilized to measure classifier performance, and in order to better understand AUC and ROC behavior, pseudo-ROC curves created from simulated data are introduced. ROC curves depend entirely upon the rankings created by classifiers. The rank distributions discussed in this paper display classifier performance in a novel form, and the behavior of these rank distributions provides insight into classifier fusion for the SCP. Rank distributions, which illustrate the probability of a particular rank containing a positive or negative instance, will be introduced and used to explain why synergistic classifier fusion occurs.

Fuzzy ROC curves for unsupervised nonparametric ensemble techniques

This paper explores a novel ensemble technique for unsupervised classification using nonparametric statistics. Multiple classification systems (MCS), or ensemble techniques, involve considering several classification methods or multiple outputs from the same method and devising techniques to reach a decision. The performance of a binary classification system can be measured on a receiver operating characteristic (ROC) curve, and the area under the curve (AUC) is exactly the Wilcoxon rank sum or Mann-Whitney U statistic, both of which are nonparametric statistics based upon ranked data. Successful performance of an unsupervised ensemble can be measured through the AUC, and the performance of different aggregation techniques for the combination of the multiple classification system decision values, or rankings in this paper, is illustrated. Aggregation techniques are based upon fuzzy logic theory, creating the fuzzy ROC curve. The one-class SVM is utilized for the unsupervised classification.