Chris J. Harris

Sparse modeling using orthogonal forward regression with PRESS statistic and regularization

The paper introduces an efficient construction algorithm for obtaining sparse linear-in-the-weights regression models based on an approach of directly optimizing model generalization capability. This is achieved by utilizing the delete-1 cross validation concept and the associated leave-one-out test error also known as the predicted residual sums of squares (PRESS) statistic, without resorting to any other validation data set for model evaluation in the model construction process. Computational efficiency is ensured using an orthogonal forward regression, but the algorithm incrementally minimizes the PRESS statistic instead of the usual sum of the squared training errors. A local regularization method can naturally be incorporated into the model selection procedure to further enforce model sparsity. The proposed algorithm is fully automatic, and the user is not required to specify any criterion to terminate the model construction procedure. Comparisons with some of the existing state-of-art modeling methods are given, and several examples are included to demonstrate the ability of the proposed algorithm to effectively construct sparse models that generalize well.

Recognising humans by gait via parametric canonical space

Abstract Based on principal component analysis (PCA), eigenspace transformation (EST) was demonstrated to be a potent metric in automatic face recognition and gait analysis by template matching, but without using data analysis to increase classification capability. Gait is a new biometric aimed to recognise subjects by the way they walk. In this article, we propose a new approach which combines canonical space transformation (CST) based on Canonical Analysis (CA), with EST for feature extraction. This method can be used to reduce data dimensionality and to optimise the class separability of different gait classes simultaneously. Each image template is projected from the high-dimensional image space to a low-dimensional canonical space. Using template matching, recognition of human gait becomes much more accurate and robust in this new space. Experimental results on a small database show how subjects can be recognised with 100% accuracy by their gait, using this method.

Model selection approaches for non-linear system identification: a review

The identification of non-linear systems using only observed finite datasets has become a mature research area over the last two decades. A class of linear-in-the-parameter models with universal approximation capabilities have been intensively studied and widely used due to the availability of many linear-learning algorithms and their inherent convergence conditions. This article presents a systematic overview of basic research on model selection approaches for linear-in-the-parameter models. One of the fundamental problems in non-linear system identification is to find the minimal model with the best model generalisation performance from observational data only. The important concepts in achieving good model generalisation used in various non-linear system-identification algorithms are first reviewed, including Bayesian parameter regularisation and models selective criteria based on the cross validation and experimental design. A significant advance in machine learning has been the development of the support vector machine as a means for identifying kernel models based on the structural risk minimisation principle. The developments on the convex optimisation-based model construction algorithms including the support vector regression algorithms are outlined. Input selection algorithms and on-line system identification algorithms are also included in this review. Finally, some industrial applications of non-linear models are discussed.

Some Remarks on Kalman Filters for the Multisensor Fusion

Abstract Multisensor data fusion has found widespread application in industry and commerce. The purpose of data fusion is to produce an improved model or estimate of a system from a set of independent data sources. There are various multisensor data fusion approaches, of which Kalman filtering is one of the most significant. Methods for Kalman filter based data fusion includes measurement fusion and state fusion. This paper gives first a simple a review of both measurement fusion and state fusion, and secondly proposes two new methods of state fusion based on fusion procedures at the prediction and update level, respectively, of the Kalman filter. The theoretical derivation for these algorithms are derived. To illustrate their application, a simple example is performed to evaluate the proposed methods and compare their performance with the conventional state fusion method and measurement fusion methods.

A Kernel-Based Two-Class Classifier for Imbalanced Data Sets

Many kernel classifier construction algorithms adopt classification accuracy as performance metrics in model evaluation. Moreover, equal weighting is often applied to each data sample in parameter estimation. These modeling practices often become problematic if the data sets are imbalanced. We present a kernel classifier construction algorithm using orthogonal forward selection (OFS) in order to optimize the model generalization for imbalanced two-class data sets. This kernel classifier identification algorithm is based on a new regularized orthogonal weighted least squares (ROWLS) estimator and the model selection criterion of maximal leave-one-out area under curve (LOO-AUC) of the receiver operating characteristics (ROCs). It is shown that, owing to the orthogonalization procedure, the LOO-AUC can be calculated via an analytic formula based on the new regularized orthogonal weighted least squares parameter estimator, without actually splitting the estimation data set. The proposed algorithm can achieve minimal computational expense via a set of forward recursive updating formula in searching model terms with maximal incremental LOO-AUC value. Numerical examples are used to demonstrate the efficacy of the algorithm

Vehicle detection and recognition in greyscale imagery

Abstract This paper details a novel two-stage vehicle detection and recognition algorithm by combining an image-processing region of interest (ROI) designator to cue a secondary recognition process implemented using principal component analysis (PCA) as input to a Multi-Layered Perceptron (MLP) classifier. Both the image-processing system and the MLP classifier have been designed for real-time implementation and data-fusion with other information sources.

Sparse kernel regression modeling using combined locally regularized orthogonal least squares and D-optimality experimental design

The note proposes an efficient nonlinear identification algorithm by combining a locally regularized orthogonal least squares (LROLS) model selection with a D-optimality experimental design. The proposed algorithm aims to achieve maximized model robustness and sparsity via two effective and complementary approaches. The LROLS method alone is capable of producing a very parsimonious model with excellent generalization performance. The D-optimality design criterion further enhances the model efficiency and robustness. An added advantage is that the user only needs to specify a weighting for the D-optimality cost in the combined model selecting criterion and the entire model construction procedure becomes automatic. The value of this weighting does not influence the model selection procedure critically and it can be chosen with ease from a wide range of values.

A Probabilistic Framework for SVM Regression and Error Bar Estimation

In this paper, we elaborate on the well-known relationship between Gaussian Processes (GP) and Support Vector Machines (SVM) under some convex assumptions for the loss functions. This paper concentrates on the derivation of the evidence and error bar approximation for regression problems. An error bar formula is derived based on the ∈-insensitive loss function.

On the modelling of nonlinear dynamic systems using support vector neural networks

Abstract Though neural networks have the ability to approximate nonlinear functions with arbitrary accuracy, good generalization results are obtained only if the structure of the network is suitably chosen. Therefore, selecting the ‘best’ structure of the neural networks is an important problem. Support vector neural networks (SVNN) are proposed in this paper, which can provide a solution to this problem. The structure of the SVNN is obtained by a constrained minimization for a given error bound similar to that in the support vector regression (SVR). After the structure is selected, its weights are computed by the linear least squares method, as it is a linear-in-weight network. Consequently, in contrast to the SVR, the output of the SVNN is unbiased. It is further shown here that the variance of the modelling error of the SVNN is bounded by the square of the given error bound in selecting its structure, and is smaller than that of the SVR. The performance of the SVNN is illustrated by a simulation example involving a benchmark nonlinear system.

Extracting human gait signatures by body segment properties

We describe a new method for extracting human gait signatures by topological analysis, using properties of body segments. The gait signature is extracted in three stages: extraction of the body contour by a thresholding and morphological filter; extraction of the leg angles based on regression analysis of contour data; finding the body points guided by known anatomical knowledge. A 2D stick figure is used to represent the human body model and trajectory-based kinematic features are extracted from the image sequences for describing and analyzing the gait motion. Also, the inherent periodicity in gait motion is analyzed by delay coordinates and a phase-space portrait. The utility of the proposed method is demonstrated in experiments, with comparison to medical data.