Carlos Alzate

Multiway Spectral Clustering with Out-of-Sample Extensions through Weighted Kernel PCA

A new formulation for multiway spectral clustering is proposed. This method corresponds to a weighted kernel principal component analysis (PCA) approach based on primal-dual least-squares support vector machine (LS-SVM) formulations. The formulation allows the extension to out-of-sample points. In this way, the proposed clustering model can be trained, validated, and tested. The clustering information is contained on the eigendecomposition of a modified similarity matrix derived from the data. This eigenvalue problem corresponds to the dual solution of a primal optimization problem formulated in a high-dimensional feature space. A model selection criterion called the balanced line fit (BLF) is also proposed. This criterion is based on the out-of-sample extension and exploits the structure of the eigenvectors and the corresponding projections when the clusters are well formed. The BLF criterion can be used to obtain clustering parameters in a learning framework. Experimental results with difficult toy problems and image segmentation show improved performance in terms of generalization to new samples and computation times.

Kernel Component Analysis Using an Epsilon-Insensitive Robust Loss Function

Kernel principal component analysis (PCA) is a technique to perform feature extraction in a high-dimensional feature space, which is nonlinearly related to the original input space. The kernel PCA formulation corresponds to an eigendecomposition of the kernel matrix: eigenvectors with large eigenvalues correspond to the principal components in the feature space. Starting from the least squares support vector machine (LS-SVM) formulation to kernel PCA, we extend it to a generalized form of kernel component analysis (KCA) with a general underlying loss function made explicit. For classical kernel PCA, the underlying loss function is L 2 . In this generalized form, one can plug in also other loss functions. In the context of robust statistics, it is known that the L 2 loss function is not robust because its influence function is not bounded. Therefore, outliers can skew the solution from the desired one. Another issue with kernel PCA is the lack of sparseness: the principal components are dense expansions in terms of kernel functions. In this paper, we introduce robustness and sparseness into kernel component analysis by using an epsilon-insensitive robust loss function. We propose two different algorithms. The first method solves a set of nonlinear equations with kernel PCA as starting points. The second method uses a simplified iterative weighting procedure that leads to solving a sequence of generalized eigenvalue problems. Simulations with toy and real-life data show improvements in terms of robustness together with a sparse representation.

Primal and dual model representations in kernel-based learning

Abstract: This paper discusses the role of primal and (Lagrange) dual model representations in problems of supervised and unsupervised learning. The specification of the estimation problem is conceived at the primal level as a constrained optimization problem. The constraints relate to the model which is expressed in terms of the feature map. From the conditions for optimality one jointly finds the optimal model representation and the model estimate. At the dual level the model is expressed in terms of a positive definite kernel function, which is characteristic for a support vector machine methodology. It is discussed how least squares support vector machines are playing a central role as core models across problems of regression, classification, principal component analysis, spectral clustering, canonical correlation analysis, dimensionality reduction and data visualization.

LS-SVM based spectral clustering and regression for predicting maintenance of industrial machines

Abstract Accurate prediction of forthcoming faults in modern industrial machines plays a key role in reducing production arrest, increasing the safety of plant operations, and optimizing manufacturing costs. The most effective condition monitoring techniques are based on the analysis of historical process data. In this paper we show how Least Squares Support Vector Machines (LS-SVMs) can be used effectively for early fault detection in an online fashion. Although LS-SVMs are existing artificial intelligence methods, in this paper the novelty is represented by their successful application to a complex industrial use case, where other approaches are commonly used in practice. In particular, in the first part we present an unsupervised approach that uses Kernel Spectral Clustering (KSC) on the sensor data coming from a vertical form seal and fill (VFFS) machine, in order to distinguish between normal operating condition and abnormal situations. Basically, we describe how KSC is able to detect in advance the need of maintenance actions in the analysed machine, due the degradation of the sealing jaws. In the second part we illustrate a nonlinear auto-regressive (NAR) model, thus a supervised learning technique, in the LS-SVM framework. We show that we succeed in modelling appropriately the degradation process affecting the machine, and we are capable to accurately predict the evolution of dirt accumulation in the sealing jaws.

Improved Electricity Load Forecasting via Kernel Spectral Clustering of Smart Meters

This paper explores kernel spectral clustering methods to improve forecasts of aggregated electricity smart meter data. The objective is to cluster the data in such a way that building a forecasting models separately for each cluster and taking the sum of forecasts leads to a better accuracy than building one forecasting model for the total aggregate of all meters. To measure the similarity between time series, we consider wavelet feature extraction and several positive-definite kernels. To forecast the aggregated meter data, we use a periodic autoregressive model with calendar and temperature information as exogenous variable. The data used in the experiments are smart meter recordings from 6,000 residential customers and small-to-medium enterprises collected by the Irish Commission for Energy Regulation (CER). The results show a 20% improvement in forecasting accuracy, where the highest gain is obtained using a kernel with the Spearmans distance. The resulting clusters show distinctive patterns particularly during hours of peak demand.

A regularized kernel CCA contrast function for ICA

A new kernel based contrast function for independent component analysis (ICA) is proposed. This criterion corresponds to a regularized correlation measure in high dimensional feature spaces induced by kernels. The formulation is a multivariate extension of the least squares support vector machine (LS-SVM) formulation to kernel canonical correlation analysis (CCA). The regularization is incorporated naturally in the primal problem leading to a dual generalized eigenvalue problem. The smallest generalized eigenvalue is a measure of correlation in the feature space and a measure of independence in the input space. Due to the primal-dual nature of the proposed approach, the measure of independence can also be extended to out-of-sample points which is important for model selection ensuring statistical reliability of the proposed measure. Computational issues due to the large size of the matrices involved in the eigendecomposition are tackled via the incomplete Cholesky factorization. Simulations with toy data, images and speech signals show improved performance on the estimation of independent components compared with existing kernel-based contrast functions.

Multiclass Semisupervised Learning Based Upon Kernel Spectral Clustering

This paper proposes a multiclass semisupervised learning algorithm by using kernel spectral clustering (KSC) as a core model. A regularized KSC is formulated to estimate the class memberships of data points in a semisupervised setting using the one-versus-all strategy while both labeled and unlabeled data points are present in the learning process. The propagation of the labels to a large amount of unlabeled data points is achieved by adding the regularization terms to the cost function of the KSC formulation. In other words, imposing the regularization term enforces certain desired memberships. The model is then obtained by solving a linear system in the dual. Furthermore, the optimal embedding dimension is designed for semisupervised clustering. This plays a key role when one deals with a large number of clusters.

Hierarchical kernel spectral clustering

Kernel spectral clustering fits in a constrained optimization framework where the primal problem is expressed in terms of high-dimensional feature maps and the dual problem is expressed in terms of kernel evaluations. An eigenvalue problem is solved at the training stage and projections onto the eigenvectors constitute the clustering model. The formulation allows out-of-sample extensions which are useful for model selection in a learning setting. In this work, we propose a methodology to reveal the hierarchical structure present on the data. During the model selection stage, several clustering model parameters leading to good clusterings can be found. These results are then combined to display the underlying cluster hierarchies where the optimal depth of the tree is automatically determined. Simulations with toy data and real-life problems show the benefits of the proposed approach.

A Weighted Kernel PCA Formulation with Out-of-Sample Extensions for Spectral Clustering Methods

A new formulation to spectral clustering methods based on the weighted kernel principal component analysis is presented. This formulation fits in the Least Squares Support Vector Machines (LS-SVM) framework as a primal-dual interpretation in the context of constrained optimization problems. Starting from the LS-SVM formulation to kernel PCA, a weighted approach is derived. An advantage of this method is the possibility to apply the trained clustering model to out-of-sample (test) data points without using approximation techniques such as the Nystrom method. Links with some existing spectral clustering techniques are given, showing that these techniques are particular cases of weighted kernel PCA. Simulation results with toy and real-life data show improvements in terms of generalization to new samples.

Image Segmentation using a Weighted Kernel PCA Approach to Spectral Clustering

In classical graph-based image segmentation, a data-driven matrix is constructed representing similarities between every pair of pixels. The eigenvectors of such matrices contain relevant information about the clusters present on the image. An approach to image segmentation using spectral clustering with out-of-sample extensions is presented. This approach is based on the weighted kernel PCA framework. An advantage of the proposed method is the possibility to train and validate the clustering model on subsampled parts of the image to be segmented. The cluster indicators for the remaining pixels can then be inferred using the out-of-sample extension. This subsampling scheme can be used to reduce the computation time of the segmentation. Simulation results with grayscale and color images show improvements in terms of computation times together with visually appealing clusters