Wafa Bel Haj Ali

Boosting k-NN for Categorization of Natural Scenes

The k-nearest neighbors (k-NN) classification rule has proven extremely successful in countless many computer vision applications. For example, image categorization often relies on uniform voting among the nearest prototypes in the space of descriptors. In spite of its good generalization properties and its natural extension to multi-class problems, the classic k-NN rule suffers from high variance when dealing with sparse prototype datasets in high dimensions. A few techniques have been proposed in order to improve k-NN classification, which rely on either deforming the nearest neighborhood relationship by learning a distance function or modifying the input space by means of subspace selection. From the computational standpoint, many methods have been proposed for speeding up nearest neighbor retrieval, both for multidimensional vector spaces and nonvector spaces induced by computationally expensive distance measures.In this paper, we propose a novel boosting approach for generalizing the k-NN rule, by providing a new k-NN boosting algorithm, called UNN (Universal Nearest Neighbors), for the induction of leveragedk-NN. We emphasize that UNN is a formal boosting algorithm in the original boosting terminology. Our approach consists in redefining the voting rule as a strong classifier that linearly combines predictions from the k closest prototypes. Therefore, the k nearest neighbors examples act as weak classifiers and their weights, called leveraging coefficients, are learned by UNN so as to minimize a surrogate risk, which upper bounds the empirical misclassification rate over training data. These leveraging coefficients allows us to distinguish the most relevant prototypes for a given class. Indeed, UNN does not affect the k-nearest neighborhood relationship, but rather acts on top of k-NN search.We carried out experiments comparing UNN to k-NN, support vector machines (SVM) and AdaBoost on categorization of natural scenes, using state-of-the art image descriptors (Gist and Bag-of-Features) on real images from Oliva and Torralba (Int. J. Comput. Vis. 42(3):145–175, 2001), Fei-Fei and Perona (IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 524–531, 2005), and Xiao et al. (IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485–3492, 2010). Results display the ability of UNN to compete with or beat the other contenders, while achieving comparatively small training and testing times.

Boosting nearest neighbors for the efficient estimation of posteriors

It is an admitted fact that mainstream boosting algorithms like AdaBoost do not perform well to estimate class conditional probabilities. In this paper, we analyze, in the light of this problem, a recent algorithm, unn, which leverages nearest neighbors while minimizing a convex loss. Our contribution is threefold. First, we show that there exists a subclass of surrogate losses, elsewhere called balanced, whose minimization brings simple and statistically efficient estimators for Bayes posteriors. Second, we show explicit convergence rates towards these estimators for unn, for any such surrogate loss, under a Weak Learning Assumption which parallels that of classical boosting results. Third and last, we provide experiments and comparisons on synthetic and real datasets, including the challenging SUN computer vision database. Results clearly display that boosting nearest neighbors may provide highly accurate estimators, sometimes more than a hundred times more accurate than those of other contenders like support vector machines.

Boosting Stochastic Newton with Entropy Constraint for Large-Scale Image Classification

Large scale image classification requires efficient scalable learning methods with linear complexity in the number of samples. Although Stochastic Gradient Descent is an efficient alternative to classical Support Vector Machine, this method suffers from slow convergence. In this paper, our contribution is two folds. First we consider the minimization of specific calibrated losses, for which we show how to reliably estimate posteriors, binary entropy and margin. Secondly we propose a Boosting Stochastic Newton Descent (BSN) method for minimization in the primal space of these specific calibrated loss. BSN approximates the inverse Hessian by the best low-rank approximation. The originality of BSN relies on the fact that it does perform a boosting scheme without computing iterative weight update over the examples. We validate BSN by benchmarking it against several variants of the state-of-the-art SGD algorithm on the large scale Image Net dataset. The results on Image Net large scale image classification display that BSN improves significantly accuracy of the SGD baseline while being faster by orders of magnitude.

Boosting k-Nearest Neighbors Classification

A major drawback of the k-nearest neighbors (k-NN) rule is the high variance when dealing with sparse prototype datasets in high dimensions. Most techniques proposed for improving k-NN classification rely either on deforming the k-NN relationship by learning a distance function or modifying the input space by means of subspace selection. Here we propose a novel boosting approach for generalizing the k-NN rule. Namely, we redefine the voting rule as a strong classifier that linearly combines predictions from the k closest prototypes. Our algorithm, called UNN (Universal Nearest Neighbors), rely on the k-nearest neighbors examples as weak classifiers and learn their weights so as to minimize a surrogate risk. These weights, called leveraging coefficients, allow us to distinguish the most relevant prototypes for a given class. Results obtained on several scene categorization datasets display the ability of UNN to compete with or beat state-of-the-art methods, while achieving comparatively small training and testing times.

Biomedical Images Classification by Universal Nearest Neighbours Classifier Using Posterior Probability

Universal Nearest Neighbours (unn) is a classifier recently proposed, which can also effectively estimates the posterior probability of each classification act. This algorithm, intrinsically binary, requires the use of a decomposition method to cope with multiclass problems, thus reducing their complexity in less complex binary subtasks. Then, a reconstruction rule provides the final classification. In this paper we show that the application of unn algorithm in conjunction with a reconstruction rule based on the posterior probabilities provides a classification scheme robust among different biomedical image datasets. To this aim, we compare unn performance with those achieved by Support Vector Machine with two different kernels and by a k Nearest Neighbours classifier, and applying two different reconstruction rules for each of the aforementioned classification paradigms. The results on one private and five public biomedical datasets show satisfactory performance.

Gentle Nearest Neighbors Boosting over Proper Scoring Rules.

Tailoring nearest neighbors algorithms to boosting is an important problem. Recent papers study an approach, UNN, which provably minimizes particular convex surrogates under weak assumptions. However, numerical issues make it necessary to experimentally tweak parts of the UNN algorithm, at the possible expense of the algorithms convergence and performance. In this paper, we propose a lightweight Newton-Raphson alternative optimizing proper scoring rules from a very broad set, and establish formal convergence rates under the boosting framework that compete with those known for UNN. To the best of our knowledge, no such boosting-compliant convergence rates were previously known in the popular Gentle Adaboosts lineage. We provide experiments on a dozen domains, including Caltech and SUN computer vision databases, comparing our approach to major families including support vector machines, (Ada)boosting and stochastic gradient descent. They support three major conclusions: (i) GNNB significantly outperforms UNN, in terms of convergence rate and quality of the outputs, (ii) GNNB performs on par with or better than computationally intensive large margin approaches, (iii) on large domains that rule out those latter approaches for computational reasons, GNNB provides a simple and competitive contender to stochastic gradient descent. Experiments include a divide-and-conquer improvement of GNNB exploiting the link with proper scoring rules optimization.

Hybrid weighted-stego detection using machine learning

This paper deals with stego-image steganalysis to detect hidden information in natural images. Hidden bits are embedded by using the Least Significant Bit (LSB) replacement mechanism. We address the problem of learning the weights which characterize the structure and the performance of the standard Weighted Stego-image (WS) detector. In this paper we propose a new Hybrid Weighted Stego-detection (HWS) algorithm. We assume that the WS weights are related to the image pixels variance through an unknown function which is decomposed onto a set of known basis functions. This yields a linear detector which consists of a linear combination of parametric features derived from the structure of the standard WS detector. The coefficients of the linear combination are learnt by minimizing calibrated losses using stochastic gradient descent or a more efficient stochastic Newton descent approach. Thus, the HWS algorithm benefits from two fundamental advantages: the posterior probability of detection is well estimated and the numerical complexity of the algorithm is linear with the number of samples and the dimension of the features. The benchmark on real images shows that HWS method outperforms standard WS baseline method.

From descriptor to boosting: Optimizing the k-NN classification rule

The k-nearest neighbor (K-NN) framework was successfully used for tasks of computer vision. In image categorization, k-NN is an important and significant rule. However, two major problems usually affect this rule: the NN classifier vote and the metric employed to compute the distance between neighbors. This paper deals with both. First, a boosting k-NN algorithm learns the coefficients of weak classifiers, hence allowing to assign weights for k-NN votes. Second, we have recourse to metric learning: a function is trained on sets of similar and dissimilar samples to increase inter-class distances and reduce intra-class ones.