Kris De Brabanter

Approximate Confidence and Prediction Intervals for Least Squares Support Vector Regression

Bias-corrected approximate 100(1-α)% pointwise and simultaneous confidence and prediction intervals for least squares support vector machines are proposed. A simple way of determining the bias without estimating higher order derivatives is formulated. A variance estimator is developed that works well in the homoscedastic and heteroscedastic case. In order to produce simultaneous confidence intervals, a simple Šidák correction and a more involved correction (based on upcrossing theory) are used. The obtained confidence intervals are compared to a state-of-the-art bootstrap-based method. Simulations show that the proposed method obtains similar intervals compared to the bootstrap at a lower computational cost.

Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes

It has been shown that Kernel Based Regression (KBR) with a least squares loss has some undesirable properties from robustness point of view. KBR with more robust loss functions, e.g. Huber or logistic losses, often give rise to more complicated computations. In this work the practical consequences of this sensitivity are explained, including the breakdown of Support Vector Machines (SVM) and weighted Least Squares Support Vector Machines (LS-SVM) for regression. In classical statistics, robustness is improved by reweighting the original estimate. We study the influence of reweighting the LS-SVM estimate using four different weight functions. Our results give practical guidelines in order to choose the weights, providing robustness and fast convergence. It turns out that Logistic and Myriad weights are suitable reweighting schemes when outliers are present in the data. In fact, the Myriad shows better performance over the others in the presence of extreme outliers (e.g. Cauchy distributed errors). These findings are then illustrated on toy example as well as on a real life data sets.

Sparse conjugate directions pursuit with application to fixed-size kernel models

This work studies an optimization scheme for computing sparse approximate solutions of over-determined linear systems. Sparse Conjugate Directions Pursuit (SCDP) aims to construct a solution using only a small number of nonzero (i.e. nonsparse) coefficients. Motivations of this work can be found in a setting of machine learning where sparse models typically exhibit better generalization performance, lead to fast evaluations, and might be exploited to define scalable algorithms. The main idea is to build up iteratively a conjugate set of vectors of increasing cardinality, in each iteration solving a small linear subsystem. By exploiting the structure of this conjugate basis, an algorithm is found (i) converging in at most D iterations for D-dimensional systems, (ii) with computational complexity close to the classical conjugate gradient algorithm, and (iii) which is especially efficient when a few iterations suffice to produce a good approximation. As an example, the application of SCDP to Fixed-Size Least Squares Support Vector Machines (FS-LSSVM) is discussed resulting in a scheme which efficiently finds a good model size for the FS-LSSVM setting, and is scalable to large-scale machine learning tasks. The algorithm is empirically verified in a classification context. Further discussion includes algorithmic issues such as component selection criteria, computational analysis, influence of additional hyper-parameters, and determination of a suitable stopping criterion.

Wavelet Filter Design for Pavement Roughness Analysis.

Control and characterization of pavement roughness is a major quality assurance requirement. With emerging technologies in real-time monitoring and increasingly stringent requirements to minimize localized roughness features, there is an opportunity to improve upon the traditional quarter-car QC algorithm used to qualify roughness. Current methods suffer from phase lag that mislocates roughness features and require relatively long profiles to achieve high accuracy. In this study, continuous and discrete wavelet bases were modified in the frequency domain to design 116 new QC-wavelet filters in the spatial domain that were used to analyze 30 road profiles. QC-wavelet filters were compared to the currently used finite difference algorithm and filtering in the frequency domain. QC-wavelet filters design based on a Daubechies and nonanalytic Morlet i.e., db21 and morl0 wavelets outperformed the other filters and algorithms in terms of characterizing overall profiles and accurately quantifying localized features. The major advantages of the new approach include accurately estimating the position and severity of localized feature, and accurately analyzing short profile segments i.e., <7.62 m.

Predicting breast cancer using an expression values weighted clinical classifier

BackgroundClinical data, such as patient history, laboratory analysis, ultrasound parameters-which are the basis of day-to-day clinical decision support-are often used to guide the clinical management of cancer in the presence of microarray data. Several data fusion techniques are available to integrate genomics or proteomics data, but only a few studies have created a single prediction model using both gene expression and clinical data. These studies often remain inconclusive regarding an obtained improvement in prediction performance. To improve clinical management, these data should be fully exploited. This requires efficient algorithms to integrate these data sets and design a final classifier.LS-SVM classifiers and generalized eigenvalue/singular value decompositions are successfully used in many bioinformatics applications for prediction tasks. While bringing up the benefits of these two techniques, we propose a machine learning approach, a weighted LS-SVM classifier to integrate two data sources: microarray and clinical parameters.ResultsWe compared and evaluated the proposed methods on five breast cancer case studies. Compared to LS-SVM classifier on individual data sets, generalized eigenvalue decomposition (GEVD) and kernel GEVD, the proposed weighted LS-SVM classifier offers good prediction performance, in terms of test area under ROC Curve (AUC), on all breast cancer case studies.ConclusionsThus a clinical classifier weighted with microarray data set results in significantly improved diagnosis, prognosis and prediction responses to therapy. The proposed model has been shown as a promising mathematical framework in both data fusion and non-linear classification problems.

New bandwidth selection criterion for Kernel PCA: Approach to dimensionality reduction and classification problems

BackgroundDNA microarrays are potentially powerful technology for improving diagnostic classification, treatment selection, and prognostic assessment. The use of this technology to predict cancer outcome has a history of almost a decade. Disease class predictors can be designed for known disease cases and provide diagnostic confirmation or clarify abnormal cases. The main input to this class predictors are high dimensional data with many variables and few observations. Dimensionality reduction of these features set significantly speeds up the prediction task. Feature selection and feature transformation methods are well known preprocessing steps in the field of bioinformatics. Several prediction tools are available based on these techniques.ResultsStudies show that a well tuned Kernel PCA (KPCA) is an efficient preprocessing step for dimensionality reduction, but the available bandwidth selection method for KPCA was computationally expensive. In this paper, we propose a new data-driven bandwidth selection criterion for KPCA, which is related to least squares cross-validation for kernel density estimation. We propose a new prediction model with a well tuned KPCA and Least Squares Support Vector Machine (LS-SVM). We estimate the accuracy of the newly proposed model based on 9 case studies. Then, we compare its performances (in terms of test set Area Under the ROC Curve (AUC) and computational time) with other well known techniques such as whole data set + LS-SVM, PCA + LS-SVM, t-test + LS-SVM, Prediction Analysis of Microarrays (PAM) and Least Absolute Shrinkage and Selection Operator (Lasso). Finally, we assess the performance of the proposed strategy with an existing KPCA parameter tuning algorithm by means of two additional case studies.ConclusionWe propose, evaluate, and compare several mathematical/statistical techniques, which apply feature transformation/selection for subsequent classification, and consider its application in medical diagnostics. Both feature selection and feature transformation perform well on classification tasks. Due to the dynamic selection property of feature selection, it is hard to define significant features for the classifier, which predicts classes of future samples. Moreover, the proposed strategy enjoys a distinctive advantage with its relatively lesser time complexity.

Spatial pavement roughness from stationary laser scanning

Abstract Pavement roughness is a key parameter for controlling pavement construction processes and for assessing ride quality during the life of a pavement system. This paper describes algorithms used in processing three-dimensional (3D) stationary terrestrial laser scanning (STLS) point clouds to obtain surface maps of point wise indices that characterise pavement roughness. The backbone of the analysis is a quarter-car model simulation over a spatial 3D mesh grid representing the pavement surface. With the rich data-set obtained by 3D scanning, the algorithms identify several dynamic responses and inferences (suspension, acceleration and jerk) at each point in the domain. Variability in the indices is compared for a ‘rough’ pavement and a ‘smooth’ pavement in the spatial domain for different speed simulations of the quarter-car model. Results show high spatial variability in the various roughness indices both longitudinally and transversely (i.e. different wheel path positions). It is proposed that pavement roughness characterisation using a spatial framework coupled with univariate statistics provides more details on the severity and location of pavement roughness features compared to the (1D) one-dimensional methods. This paper describes approaches that provide an algorithmic framework for others collecting similar STLS 3D spatial data to be used in advanced pavement roughness characterisation.

Quantifying Roughness of Unpaved Roads by Terrestrial Laser Scanning

Unpaved roads represent a major component of the roadway network in Iowa and require regular surface grading to control roughness (e.g., corrugation). This paper presents a study that shows how terrestrial laser scanning can characterize road roughness in relation to (a) a spatially analyzed international roughness index (IRI), (b) a fast Fourier transform (FFT) spectral analysis, and (c) surface texture characterization that uses statistical analysis. Algorithms are presented for each of the roughness calculations. The spatial nature of terrestrial laser scanning makes possible the determination of IRI values along any selected path within the scan area. In this study, median IRI values were used to characterize overall road roughness. To characterize the nature of road roughness, FFT was used to decompose the laser scan height spectrum. Examples are presented for unpaved road sections with a relatively smooth surface, a smooth surface with corrugations, an unsystematically rough surface, and an unsystematically rough surface with corrugations. On the basis of the observation that the surface aggregate materials were segregated, selected small regions were studied with a scan with spacing of 0.5-mm points. These scans were filtered and statistically analyzed for variations in height by using root mean square, skewness, and kurtosis parameters. These parameters were studied in relation to the values for particle size index for the surface materials of the three distinct gradations. The algorithms presented will be of value to terrestrial laser scanning users interested in characterizing unpaved roadway conditions.

Smoothed Nonparametric Derivative Estimation Based on Weighted Difference Sequences

We present a simple but effective fully automated framework for estimating derivatives nonparametrically based on weighted difference sequences. Although regression estimation is often studied more, derivative estimation is of equal importance. For example in the study of exploration of structures in curves, comparison of regression curves, analysis of human growth data, etc. Via the introduced weighted difference sequence, we approximate the true derivative and create a new data set which can be smoothed by any nonparametric regression estimator. However, the new data sets created by this technique are no longer independent and identically distributed (i.i.d.) random variables. Due to the non-i.i.d. nature of the data, model selection methods tend to produce bandwidths (or smoothing parameters) which are too small. In this paper, we propose a method based on bimodal kernels to cope with the non-i.i.d. data in the local polynomial regression framework.

Robustness of kernel based regression: Influence and weight functions

It has been shown that kernel based regression (KBR) with a least squares loss has some undesirable properties from robustness point of view. KBR with more robust loss functions, e.g. Huber or Logistic losses, often give rise to more complicated computations. In classical statistics, robustness is improved by reweighting the original estimate. We study the influence of reweighting the LS-KBR estimate using three well-known weight functions and one new weight function called Myriad. Our results give practical guidelines in order to choose the weights, providing robustness and fast convergence. It turns out that Logistic and Myriad weights are suitable reweighting schemes when outliers are present in the data. In fact, the Myriad shows better performance over the others in the presence of extreme outliers (e.g. Cauchy distributed errors). These findings are then illustrated on toy example as well as on a real life data sets. Finally, we establish an empirical maxbias curve to demonstrate the ability of the proposed methodology.