André Elisseeff

An Introduction to Feature Extraction

This chapter introduces the reader to the various aspects of feature extraction covered in this book. Section 1 reviews definitions and notations and proposes a unified view of the feature extraction problem. Section 2 is an overview of the methods and results presented in the book, emphasizing novel contributions. Section 3 provides the reader with an entry point in the field of feature extraction by showing small revealing examples and describing simple but effective algorithms. Finally, Section 4 introduces a more theoretical formalism and points to directions of research and open problems.

Subagging for credit scoring models

The logistic regression framework has been for long time the most used statistical method when assessing customer credit risk. Recently, a more pragmatic approach has been adopted, where the first issue is credit risk prediction, instead of explanation. In this context, several classification techniques have been shown to perform well on credit scoring, such as support vector machines among others. While the investigation of better classifiers is an important research topic, the specific methodology chosen in real world applications has to deal with the challenges arising from the real world data collected in the industry. Such data are often highly unbalanced, part of the information can be missing and some common hypotheses, such as the i.i.d. one, can be violated. In this paper we present a case study based on a sample of IBM Italian customers, which presents all the challenges mentioned above. The main objective is to build and validate robust models, able to handle missing information, class unbalancedness and non-iid data points. We define a missing data imputation method and propose the use of an ensemble classification technique, subagging, particularly suitable for highly unbalanced data, such as credit scoring data. Both the imputation and subagging steps are embedded in a customized cross-validation loop, which handles dependencies between different credit requests. The methodology has been applied using several classifiers (kernel support vector machines, nearest neighbors, decision trees, Adaboost) and their subagged versions. The use of subagging improves the performance of the base classifier and we will show that subagging decision trees achieve better performance, still keeping the model simple and reasonably interpretable.

A partial correlation-based algorithm for causal structure discovery with continuous variables

We present an algorithm for causal structure discovery suited in the presence of continuous variables. We test a version based on partial correlation that is able to recover the structure of a recursive linear equations model and compare it to the well-known PC algorithm on large networks. PC is generally outperformed in run time and number of structural errors.

Causal networks for risk and compliance: methodology and application

This paper presents a statistical approach to quantitatively measure the current exposure of a company to failures and defects in product quality or to compliance to government regulations. This approach is based on causal networks, which have previously been applied to other fields, such as systems maintenance and reliability. Causal networks allow analysts to causally explain the values of variables (an explanatory approach), to assess the effect of interventions on the structure of the data-generating process, and to evaluate Bwhat-if[ scenarios, that is, alternative methods or policies (an exploratory approach). Building the causal structure raises some challenges. In particular, there is no automated way to collect the needed data. We present a methodology for model selection and probability elicitation based on expert knowledge. We apply the proposed approach to the case of pharmaceutical manufacturing processes. The use of such networks allows for a more rigorous comparison of practices across different manufacturing sites, creates the opportunity for risk remediation, and allows us to evaluate alternative methods and approaches.

Guest editorial: Learning theory

The theoretical study of the learning ability of machines was initiated in the sixties with theworks of Gold, Solomonoff, Vapnik and Chervonenkis among a few others. In almost halfa century, this discipline has developed into various directions and several sub-areas havebranched out. The notion of a unique theory of learning has disappeared in favor of a set ofmachine learning theories.