T. Van Gestel

Benchmarking state-of-the-art classification algorithms for credit scoring

In this paper, we study the performance of various state-of-the-art classification algorithms applied to eight real-life credit scoring data sets. Some of the data sets originate from major Benelux and UK financial institutions. Different types of classifiers are evaluated and compared. Besides the well-known classification algorithms (eg logistic regression, discriminant analysis, k-nearest neighbour, neural networks and decision trees), this study also investigates the suitability and performance of some recently proposed, advanced kernel-based classification algorithms such as support vector machines and least-squares support vector machines (LS-SVMs). The performance is assessed using the classification accuracy and the area under the receiver operating characteristic curve. Statistically significant performance differences are identified using the appropriate test statistics. It is found that both the LS-SVM and neural network classifiers yield a very good performance, but also simple classifiers such as logistic regression and linear discriminant analysis perform very well for credit scoring.

Financial time series prediction using least squares support vector machines within the evidence framework

The Bayesian evidence framework is applied in this paper to least squares support vector machine (LS-SVM) regression in order to infer nonlinear models for predicting a financial time series and the related volatility. On the first level of inference, a statistical framework is related to the LS-SVM formulation which allows one to include the time-varying volatility of the market by an appropriate choice of several hyper-parameters. The hyper-parameters of the model are inferred on the second level of inference. The inferred hyper-parameters, related to the volatility, are used to construct a volatility model within the evidence framework. Model comparison is performed on the third level of inference in order to automatically tune the parameters of the kernel function and to select the relevant inputs. The LS-SVM formulation allows one to derive analytic expressions in the feature space and practical expressions are obtained in the dual space replacing the inner product by the related kernel function using Mercers theorem. The one step ahead prediction performances obtained on the prediction of the weekly 90-day T-bill rate and the daily DAX30 closing prices show that significant out of sample sign predictions can be made with respect to the Pesaran-Timmerman test statistic.

Bayesian framework for least-squares support vector machine classifiers, Gaussian processes, and kernel fisher discriminant analysis

The Bayesian evidence framework has been successfully applied to the design of multilayer perceptrons (MLPs) in the work of MacKay. Nevertheless, the training of MLPs suffers from drawbacks like the nonconvex optimization problem and the choice of the number of hidden units. In support vector machines (SVMs) for classification, as introduced by Vapnik, a nonlinear decision boundary is obtained by mapping the input vector first in a nonlinear way to a high-dimensional kernel-induced feature space in which a linear large margin classifier is constructed. Practical expressions are formulated in the dual space in terms of the related kernel function, and the solution follows from a (convex) quadratic programming (QP) problem. In least-squares SVMs (LS-SVMs), the SVM problem formulation is modified by introducing a least-squares cost function and equality instead of inequality constraints, and the solution follows from a linear system in the dual space. Implicitly, the least-squares formulation corresponds to a regression formulation and is also related to kernel Fisher discriminant analysis. The least-squares regression formulation has advantages for deriving analytic expressions in a Bayesian evidence framework, in contrast to the classification formulations used, for example, in gaussian processes (GPs). The LS-SVM formulation has clear primal-dual interpretations, and without the bias term, one explicitly constructs a model that yields the same expressions as have been obtained with GPs for regression. In this article, the Bayesian evidence frame-work is combined with the LS-SVM classifier formulation. Starting from the feature space formulation, analytic expressions are obtained in the dual space on the different levels of Bayesian inference, while posterior class probabilities are obtained by marginalizing over the model param-eters. Empirical results obtained on 10 public domain data sets show that the LS-SVM classifier designed within the Bayesian evidence framework consistently yields good generalization performances.

A support vector machine formulation to PCA analysis and its kernel version

In this paper, we present a simple and straightforward primal-dual support vector machine formulation to the problem of principal component analysis (PCA) in dual variables. By considering a mapping to a high-dimensional feature space and application of the kernel trick (Mercer theorem), kernel PCA is obtained as introduced by Scholkopf et al. (2002). While least squares support vector machine classifiers have a natural link with the kernel Fisher discriminant analysis (minimizing the within class scatter around targets +1 and -1), for PCA analysis one can take the interpretation of a one-class modeling problem with zero target value around which one maximizes the variance. The score variables are interpreted as error variables within the problem formulation. In this way primal-dual constrained optimization problem interpretations to the linear and kernel PCA analysis are obtained in a similar style as for least square-support vector machine classifiers.

Decompositional Rule Extraction from Support Vector Machines by Active Learning

Support vector machines (SVMs) are currently state-of-the-art for the classification task and, generally speaking, exhibit good predictive performance due to their ability to model nonlinearities. However, their strength is also their main weakness, as the generated nonlinear models are typically regarded as incomprehensible black-box models. In this paper, we propose a new active learning-based approach (ALBA) to extract comprehensible rules from opaque SVM models. Through rule extraction, some insight is provided into the logics of the SVM model. ALBA extracts rules from the trained SVM model by explicitly making use of key concepts of the SVM: the support vectors, and the observation that these are typically close to the decision boundary. Active learning implies the focus on apparent problem areas, which for rule induction techniques are the regions close to the SVM decision boundary where most of the noise is found. By generating extra data close to these support vectors that are provided with a class label by the trained SVM model, rule induction techniques are better able to discover suitable discrimination rules. This performance increase, both in terms of predictive accuracy as comprehensibility, is confirmed in our experiments where we apply ALBA on several publicly available data sets.

Neural network survival analysis for personal loan data

Traditionally, credit scoring aimed at distinguishing good payers from bad payers at the time of the application. The timing when customers default is also interesting to investigate since it can provide the bank with the ability to do profit scoring. Analysing when customers default is typically tackled using survival analysis. In this paper, we discuss and contrast statistical and neural network approaches for survival analysis. Compared to the proportional hazards model, neural networks may offer an interesting alternative because of their universal approximation property and the fact that no baseline hazard assumption is needed. Several neural network survival analysis models are discussed and evaluated according to their way of dealing with censored observations, time-varying inputs, the monotonicity of the generated survival curves and their scalability. In the experimental part, we contrast the performance of a neural network survival analysis model with that of the proportional hazards model for predicting both loan default and early repayment using data from a UK financial institution.

Knowledge Discovery In A Direct Marketing Case Using Least Squares Support Vector Machines

We study the problem of repeat‐purchase modeling in a direct marketing setting using Belgian data. More specifically, we investigate the detection and qualification of the most relevant explanatory variables for predicting purchase incidence. The analysis is based on a wrapped form of input selection using a sensitivity based pruning heuristic to guide a greedy, stepwise, and backward traversal of the input space. For this purpose, we make use of a powerful and promising least squares support vector machine (LS‐SVM) classifier formulation. This study extends beyond the standard recency frequency monetary (RFM) modeling semantics in two ways: (1) by including alternative operationalizations of the RFM variables, and (2) by adding several other (non‐RFM) predictors. Results indicate that elimination of redundant/irrelevant inputs allows significant reduction of model complexity. The empirical findings also highlight the importance of frequency and monetary variables, while the recency variable category seems to be of somewhat lesser importance to the case at hand. Results also point to the added value of including non‐RFM variables for improving customer profiling. More specifically, customer/company interaction, measured using indicators of information requests and complaints, and merchandise returns provide additional predictive power to purchase incidence modeling for database marketing. © 2001 John Wiley & Sons, Inc.

Inferring descriptive and approximate fuzzy rules for credit scoring using evolutionary algorithms

Generating both accurate as well as explanatory classification rules is becoming increasingly important in a knowledge discovery context. In this paper, we investigate the power and usefulness of fuzzy classification rules for data mining purposes. We propose two evolutionary fuzzy rule learners: an evolution strategy that generates approximate fuzzy rules, whereby each rule has its own specific definition of membership functions, and a genetic algorithm that extracts descriptive fuzzy rules, where all fuzzy rules share a common, linguistically interpretable definition of membership functions in disjunctive normal form. The performance of the evolutionary fuzzy rule learners is compared with that of Nefclass, a neurofuzzy classifier, and a selection of other well-known classification algorithms on a number of publicly available data sets and two real life Benelux financial credit scoring data sets. It is shown that the genetic fuzzy classifiers compare favourably with the other classifiers in terms of classification accuracy. Furthermore, the approximate and descriptive fuzzy rules yield about the same classification accuracy across the different data sets

Preoperative prediction of malignancy of ovarian tumors using least squares support vector machines

In this work, we develop and evaluate several least squares support vector machine (LS-SVM) classifiers within the Bayesian evidence framework, in order to preoperatively predict malignancy of ovarian tumors. The analysis includes exploratory data analysis, optimal input variable selection, parameter estimation, and performance evaluation via receiver operating characteristic (ROC) curve analysis. LS-SVM models with linear and radial basis function (RBF) kernels, and logistic regression models have been built on 265 training data, and tested on 160 newly collected patient data. The LS-SVM model with nonlinear RBF kernel achieves the best performance, on the test set with the area under the ROC curve (AUC), sensitivity and specificity equal to 0.92, 81.5% and 84.0%, respectively. The best averaged performance over 30 runs of randomized cross-validation is also obtained by an LS-SVM RBF model, with AUC, sensitivity and specificity equal to 0.94, 90.0% and 80.6%, respectively. These results show that the LS-SVM models have the potential to obtain a reliable preoperative distinction between benign and malignant ovarian tumors, and to assist the clinicians for making a correct diagnosis.

Identification of stable models in subspace identification by using regularization

In subspace identification methods, the system matrices are usually estimated by least squares, based on estimated Kalman filter state sequences and the observed inputs and outputs. For a finite number of data points, the estimated system matrix is not guaranteed to be stable, even when the true linear system is known to be stable. In this paper, stability is imposed by using regularization. The regularization term used here is the trace of a matrix which involves the dynamical system matrix and a positive (semi) definite weighting matrix. The amount of regularization can be determined from a generalized eigenvalue problem. The data augmentation method of Chui and Maciejowski (1996) is obtained by using specific choices for the weighting matrix in the regularization term.