Zhanxing Zhu

Supervised Distance Preserving Projections

In this work, we consider dimensionality reduction in supervised settings and, specifically, we focus on regression problems. A novel algorithm, the supervised distance preserving projection (SDPP), is proposed. The SDPP minimizes the difference between pairwise distances among projected input covariates and distances among responses locally. This minimization of distance differences leads to the effect that the local geometrical structure of the low-dimensional subspace retrieved by the SDPP mimics that of the response space. This, not only facilitates an efficient regressor design but it also uncovers useful information for visualization. The SDPP achieves this goal by learning a linear parametric mapping and, thus, it can easily handle out-of-sample data points. For nonlinear data, a kernelized version of the SDPP is also derived. In addition, an intuitive extension of the SDPP is proposed to deal with classification problems. The experimental evaluation on both synthetic and real-world data sets demonstrates the effectiveness of the SDPP, showing that it performs comparably or superiorly to state-of-the-art approaches.

Local linear regression for soft-sensor design with application to an industrial deethanizer

Abstract Soft-sensors for estimating in real-time important quality variables are a key technology in modern process industry. The successful development of a soft-sensor whose performance does not deteriorate with time and changing process characteristics is troublesome and only seldom achieved in real-world setups. The design of soft-sensors based on local regression models is becoming popular. Simplicity of calibration, ability to handle nonlinearities and, most importantly, reduced maintenance costs while retaining the requested accuracy are the major assets. In this paper, we introduce several approaches for defining an appropriate locality neighborhood and we propose a recursive version of local linear regression for soft-sensor design. To support the presentation, we discuss the results in designing a soft-sensor for estimating the ethane concentration from the bottom of a full-scale deethanizer.

Multiplicative Updates for Learning with Stochastic Matrices

Stochastic matrices are arrays whose elements are discrete probabilities. They are widely used in techniques such as Markov Chains, probabilistic latent semantic analysis, etc. In such learning problems, the learned matrices, being stochastic matrices, are non-negative and all or part of the elements sum up to one. Conventional multiplicative updates which have been widely used for nonnegative learning cannot accommodate the stochasticity constraint. Simply normalizing the nonnegative matrix in learning at each step may have an adverse effect on the convergence of the optimization algorithm. Here we discuss and compare two alternative ways in developing multiplicative update rules for stochastic matrices. One reparameterizes the matrices before applying the multiplicative update principle, and the other employs relaxation with Lagrangian multipliers such that the updates jointly optimize the objective and steer the estimate towards the constraint manifold. We compare the new methods against the conventional normalization approach on two applications, parameter estimation of Hidden Markov Chain Model and Information-Theoretic Clustering. Empirical studies on both synthetic and real-world datasets demonstrate that the algorithms using the new methods perform more stably and efficiently than the conventional ones.

Spectroscopic monitoring of diesel fuels using Supervised Distance Preserving Projections

Abstract In this work, we discuss a recently proposed approach for supervised dimensionality reduction, the Supervised Distance Preserving Projection and, we investigate its applicability to monitoring materials properties from spectroscopic observations. Motivated by continuity preservation, the SDPP is a linear projection method where the local geometry of the points in the low-dimensional subspace mimics the geometry of the points in the response space. Such a mapping facilitates an efficient regressor design and it may also uncover useful information for visualisation. An experimental evaluation is conducted to show the performance of the SDPP and compare it with a number of state-of-the-art approaches for unsupervised and supervised dimensionality reduction. For the task, the results obtained on a benchmark problem consisting of a set of NIR spectra of diesel fuels and six different chemico-physical properties of those fuels is discussed. Based on the experimental results, the SDPP leads to accurate and parsimonious projections that can be used in the design of efficient regression models.

Automatic rank determination in projective nonnegative matrix factorization

Projective Nonnegative Matrix Factorization (PNMF) has demonstrated advantages in both sparse feature extraction and clustering. However, PNMF requires users to specify the column rank of the approximative projection matrix, the value of which is unknown before-hand. In this paper, we propose a method called ARDPNMF to automatically determine the column rank in PNMF. Our method is based on automatic relevance determination (ARD) with Jeffreys prior. After deriving the multiplicative update rule using the expectation-maximization technique for ARDPNMF, we test it on various synthetic and real-world datasets for feature extraction and clustering applications to show the effectiveness of our algorithm. For FERET faces and the Swimmer dataset, interpretable number of features are obtained correctly via our algorithm. Several UCI datasets for clustering are also tested, in which we find that ARDPNMF can estimate the number of clusters quite accurately with low deviation and good cluster purity.

Aggregation under bias: Rényi divergence aggregation and its implementation via machine learning markets

Trading in information markets, such as machine learning markets, has been shown to be an effective approach for aggregating the beliefs of different agents. In a machine learning context, aggregation commonly uses forms of linear opinion pools, or logarithmic (log) opinion pools. It is interesting to relate information market aggregation to the machine learning setting. In this paper we introduce a spectrum of compositional methods, Renyi divergence aggregators, that interpolate between log opinion pools and linear opinion pools. We show that these compositional methods are maximum entropy distributions for aggregating information from agents subject to individual biases, with the Renyi divergence parameter dependent on the bias. In the limit of no bias this reduces to the optimal limit of log opinion pools. We demonstrate this relationship practically on both simulated and real datasets. We then return to information markets and show that Renyi divergence aggregators are directly implemented by machine learning markets with isoelastic utilities, and so can result from autonomous self interested decision making by individuals contributing different predictors. The risk averseness of the isoelastic utility directly relates to the Renyi divergence parameter, and hence encodes how much an agent believes (s)he may be subject to an individual bias that could affect the trading outcome: if an agent believes (s)he might be acting on significantly biased information, a more risk averse isoelastic utility is warranted.

Monitoring Diesel Fuels with Supervised Distance Preserving Projections and Local Linear Regression

In this work, we discuss a recently proposed approach for supervised dimensionality reduction, the Supervised Distance Preserving Projection (SDPP) and, we investigate its applicability to monitoring materials properties from spectroscopic observations using Local Linear Regression (LLR). An experimental evaluation is conducted to show the performance of the SDPP and LLR and compare it with a number of state-of-the-art approaches for unsupervised and supervised dimensionality reduction. For the task, the results obtained on a benchmark problem consisting of a set of NIR spectra of diesel fuels and six different chemico-physical properties of those fuels are discussed. Based on the experimental results, the SDPP leads to accurate and parsimonious projections that can be effectively used in the design of estimation models based on local linear regression.