Predrag Neskovic

Neighborhood size selection in the k-nearest-neighbor rule using statistical confidence

The k-nearest-neighbor rule is one of the most attractive pattern classification algorithms. In practice, the choice of k is determined by the cross-validation method. In this work, we propose a new method for neighborhood size selection that is based on the concept of statistical confidence. We define the confidence associated with a decision that is made by the majority rule from a finite number of observations and use it as a criterion to determine the number of nearest neighbors needed. The new algorithm is tested on several real-world datasets and yields results comparable to the k-nearest-neighbor rule. However, in contrast to the k-nearest-neighbor rule that uses a fixed number of nearest neighbors throughout the feature space, our method locally adjusts the number of nearest neighbors until a satisfactory level of confidence is reached. In addition, the statistical confidence provides a natural way to balance the trade-off between the reject rate and the error rate by excluding patterns that have low confidence levels. We believe that this property of our method can be of great importance in applications where the confidence with which a decision is made is equally or more important than the overall error rate.

Training data selection for support vector machines

In recent years, support vector machines (SVMs) have become a popular tool for pattern recognition and machine learning. Training a SVM involves solving a constrained quadratic programming problem, which requires large memory and enormous amounts of training time for large-scale problems. In contrast, the SVM decision function is fully determined by a small subset of the training data, called support vectors. Therefore, it is desirable to remove from the training set the data that is irrelevant to the final decision function. In this paper we propose two new methods that select a subset of data for SVM training. Using real-world datasets, we compare the effectiveness of the proposed data selection strategies in terms of their ability to reduce the training set size while maintaining the generalization performance of the resulting SVM classifiers. Our experimental results show that a significant amount of training data can be removed by our proposed methods without degrading the performance of the resulting SVM classifiers.

Improving nearest neighbor rule with a simple adaptive distance measure

The k-nearest neighbor rule is one of the simplest and most attractive pattern classification algorithms. However, it faces serious challenges when patterns of different classes overlap in some regions in the feature space. In the past, many researchers developed various adaptive or discriminant metrics to improve its performance. In this paper, we demonstrate that an extremely simple adaptive distance measure significantly improves the performance of the k-nearest neighbor rule.

Selecting Data for Fast Support Vector Machines Training

In recent years, support vector machines (SVMs) have become a popular tool for pattern recognition and machine learning. Training a SVM involves solving a constrained quadratic programming problem, which requires large memory and enormous amounts of training time for largescale problems. In contrast, the SVM decision function is fully determined by a small subset of the training data, called support vectors. Therefore, it is desirable to remove from the training set the data that is irrelevant to the final decision function. In this paper we propose two new methods that select a subset of data for SVM training. Using real-world datasets, we compare the effectiveness of the proposed data selection strategies in terms of their ability to reduce the training set size while maintaining the generalization performance of the resulting SVM classifiers. Our experimental results show that a significant amount of training data can be removed by our proposed methods without degrading the performance of the resulting SVM classifiers.

Pattern classification via single spheres

Previous sphere-based classification algorithms usually need a number of spheres in order to achieve good classification performance. In this paper, inspired by the support vector machines for classification and the support vector data description method, we present a new method for constructing single spheres that separate data with the maximum separation ratio. In contrast to previous methods that construct spheres in the input space, the new method constructs separating spheres in the feature space induced by the kernel. As a consequence, the new method is able to construct a single sphere in the feature space to separate patterns that would otherwise be inseparable when using a sphere in the input space. In addition, by adjusting the ratio of the radius of the sphere to the separation margin, it can provide a series of solutions ranging from spherical to linear decision boundaries, effectively encompassing both the support vector machines for classification and the support vector data description method. Experimental results show that the new method performs well on both artificial and real-world datasets.

Bayes classification based on minimum bounding spheres

The minimum bounding sphere of a set of data, defined as the smallest sphere enclosing the data, was first used by Scholkopf et al. to estimate the VC-dimension of support vector classifiers and later applied by Tax and Duin to data domain description. Given a set of data, the minimum bounding sphere of each class can be computed by solving a quadratic programming problem. Since the spheres are constructed for each class separately, they can be used to deal with the multi-class classification problem easily, as proposed by Zhu et al. In this paper, we show that the decision rule proposed by Zhu et al. is generally insufficient to achieve the state-of-the-art classification performance. We, therefore, propose a new decision rule based on the Bayes decision theory. This new decision rule significantly improves the performance of the resulting sphere-based classifier. In addition to its low computational complexity and easy expandability to multi-class problems, the new classifier achieves comparable performance to the standard support vector machines on most of the real-world data sets being tested.

Three-dimensional shape representation from curvature dependent surface evolution

This paper presents a novel approach to surface representation based on its differential deformations. The evolution of an arbitrary curve by curvature deforms it to a round point while in the process simplifying it. Similarly, we seek a process that deforms an arbitrary surface into a sphere without developing self-intersections, in the process creating a sequence of increasingly simpler surfaces. No previously studied curvature dependent flow satisfies this requirement: mean curvature flow leads to a splitting of the surface, while Gaussian curvature flow leads to instabilities. Thus, in search for such a process, we impose constraints (motivated by visual representation) to narrow down the space of candidate flows. Our main result is to establish a direction for the movement of points to avoid self-intersections: (1) convex elliptic points should move in, while concave elliptic points move out; and (2) hyperbolic and parabolic points should not move at all. Accordingly, we propose /spl part//spl psi///spl part/t=sign(H)/spl radic/(G.<<ETX>>

A Minimum Sphere Covering Approach to Pattern Classification

In this paper we present a minimum sphere covering approach to pattern classification that seeks to construct a minimum number of spheres to represent the training data and formulate it as an integer programming problem. Using soft threshold functions, we further derive a linear programming problem whose solution gives rise to radial basis function (RBF) classifiers and sigmoid function classifiers. In contrast to traditional RBF and sigmoid function networks, in which the number of units is specified a priori, our method provides a new way to construct RBF and sigmoid function networks that explicitly minimizes the number of base units in the resulting classifiers. Our approach is advantageous compared to SVMs with Gaussian kernels in that it provides a natural construction of kernel matrices and it directly minimizes the number of basis functions. Experiments using real-world datasets demonstrate the competitiveness of our method in terms of classification performance and sparsity of the solution

An integration of online and pseudo-online information for cursive word recognition

In this paper, we present a novel method to extract stroke order independent information from online data. This information, which we term pseudo-online, conveys relevant information on the offline representation of the word. Based on this information, a combination of classification decisions from online and pseudo-online cursive word recognizers is performed to improve the recognition of online cursive words. One of the most valuable aspects of this approach with respect to similar methods that combine online and offline classifiers for word recognition is that the pseudo-online representation is similar to the online signal and, hence, word recognition is based on a single engine. Results demonstrate that the pseudo-online representation is useful as the combination of classifiers perform better than those based solely on pure online information.

A probabilistic model for cursive handwriting recognition using spatial context

In this work we introduce a probabilistic model that utilizes spatial contextual information to aid recognition when dealing with ambiguous segmentations of handwritten patterns. The recognition problem is formulated as an optimization problem in a Bayesian framework by explicitly conditioning on the spatial configuration of the letters. As a consequence, and in contrast to HMMs, the proposed model can handle duration modeling without an increase in computational complexity. We test the model on a real-world handwriting dataset and discuss several factors that affect the recognition performance.