Nello Cristianini

The Learning Methodology

The construction of machines capable of learning from experience has for a long time been the object of both philosophical and technical debate. The technical aspect of the debate has received an enormous impetus from the advent of electronic computers. They have demonstrated that machines can display a significant level of learning ability, though the boundaries of this ability are far from being clearly defined. The availability of reliable learning systems is of strategic importance, as there are many tasks that cannot be solved by classical programming techniques, since no mathematical model of the problem is available. So for example it is not known how to write a computer program to perform hand-written character recognition, though there are plenty of examples available. It is therefore natural to ask if a computer could be trained to recognise the letter ‘A’ from examples – after all this is the way humans learn to read. We will refer to this approach to problem solving as the learning methodology The same reasoning applies to the problem of finding genes in a DNA sequence, filtering email, detecting or recognising objects in machine vision, and so on. Solving each of these problems has the potential to revolutionise some aspect of our life, and for each of them machine learning algorithms could provide the key to its solution. In this chapter we will introduce the important components of the learning methodology, give an overview of the different kinds of learning and discuss why this approach has such a strategic importance. After the framework of the learning methodology has been introduced, the chapter ends with a roadmap for the rest of the book, anticipating the key themes, and indicating why Support Vector Machines meet many of the challenges confronting machine learning systems. As this roadmap will descibe the role of the different chapters, we urge our readers to refer to it before delving further into the book.

An Introduction to Support Vector Machines and Other Kernel-based Learning Methods: Kernel-Induced Feature Spaces

The limited computational power of linear learning machines was highlighted in the 1960s by Minsky and Papert. In general, complex real-world applications require more expressive hypothesis spaces than linear functions. Another way of viewing this problem is that frequently the target concept cannot be expressed as a simple linear combination of the given attributes, but in general requires that more abstract features of the data be exploited. Multiple layers of thresholded linear functions were proposed as a solution to this problem, and this approach led to the development of multi-layer neural networks and learning algorithms such as back-propagation for training such systems. Kernel representations offer an alternative solution by projecting the data into a high dimensional feature space to increase the computational power of the linear learning machines of Chapter 2. The use of linear machines in the dual representation makes it possible to perform this step implicitly. As noted in Chapter 2, the training examples never appear isolated but always in the form of inner products between pairs of examples. The advantage of using the machines in the dual representation derives from the fact that in this representation the number of tunable parameters does not depend on the number of attributes being used. By replacing the inner product with an appropriately chosen ‘kernel’ function, one can implicitly perform a non-linear mapping to a high dimensional feature space without increasing the number of tunable parameters, provided the kernel computes the inner product of the feature vectors corresponding to the two inputs. […]

Kernel Methods for Pattern Analysis: Basic kernels and kernel types

There are two key properties that are required of a kernel function for an application. Firstly, it should capture the measure of similarity appropriate to the particular task and domain, and secondly, its evaluation should require significantly less computation than would be needed in an explicit evaluation of the corresponding feature mapping ϕ. Both of these issues will be addressed in the next four chapters but the current chapter begins the consideration of the efficiency question. A number of computational methods can be deployed in order to shortcut the computation: some involve using closed-form analytic expressions, others exploit recursive relations, and others are based on sampling. This chapter aims to show several different methods in action, with the aim of illustrating how to design new kernels for specific applications. It will also pave the way for the final three chapters that carry these techniques into the design of advanced kernels. We will also return to an important theme already broached in Chapter 3, namely that kernel functions are not restricted to vectorial inputs: kernels can be designed for objects and structures as diverse as strings, graphs, text documents, sets and graph-nodes. Given the different evaluation methods and the diversity of the types of data on which kernels can be defined, together with the methods for composing and manipulating kernels outlined in Chapter 3, it should be clear how versatile this approach to data modelling can be, allowing as it does for refined customisations of the embedding map ϕ to the problem at hand.

Kernel Methods for Pattern Analysis: Basic concepts

The lectures will introduce the kernel methods approach to pattern analysis [1] through the particular example of support vector machines for classification. The presentation touches on: generalization, optimization, dual representation, kernel design and algorithmic implementations. We then broaden the discussion to consider general kernel methods by introducing different kernels, different learning tasks, and subspace methods such as kernel PCA. The emphasis is on the flexibility of the approach in applying the analyses to different data, with the caveat that the design of the kernel must rely on domain knowledge. Nonetheless we will argue that, ignoring the technical requirement of positive semi-definiteness, kernel design is not an unnatural task for a practitioner. The overall aim is to give a view of the subject that will enable newcomers to the field to gain their bearings so that they can move to apply or develop the techniques for their particular application.

Kernel Methods for Pattern Analysis: Constructing kernels

The lectures will introduce the kernel methods approach to pattern analysis [1] through the particular example of support vector machines for classification. The presentation touches on: generalization, optimization, dual representation, kernel design and algorithmic implementations. We then broaden the discussion to consider general kernel methods by introducing different kernels, different learning tasks, and subspace methods such as kernel PCA. The emphasis is on the flexibility of the approach in applying the analyses to different data, with the caveat that the design of the kernel must rely on domain knowledge. Nonetheless we will argue that, ignoring the technical requirement of positive semi-definiteness, kernel design is not an unnatural task for a practitioner. The overall aim is to give a view of the subject that will enable newcomers to the field to gain their bearings so that they can move to apply or develop the techniques for their particular application.

Kernel Methods for Pattern Analysis: Frontmatter

The lectures will introduce the kernel methods approach to pattern analysis [1] through the particular example of support vector machines for classification. The presentation touches on: generalization, optimization, dual representation, kernel design and algorithmic implementations. We then broaden the discussion to consider general kernel methods by introducing different kernels, different learning tasks, and subspace methods such as kernel PCA. The emphasis is on the flexibility of the approach in applying the analyses to different data, with the caveat that the design of the kernel must rely on domain knowledge. Nonetheless we will argue that, ignoring the technical requirement of positive semi-definiteness, kernel design is not an unnatural task for a practitioner. The overall aim is to give a view of the subject that will enable newcomers to the field to gain their bearings so that they can move to apply or develop the techniques for their particular application.

Kernel Methods for Pattern Analysis: Pattern analysis algorithms

The lectures will introduce the kernel methods approach to pattern analysis [1] through the particular example of support vector machines for classification. The presentation touches on: generalization, optimization, dual representation, kernel design and algorithmic implementations. We then broaden the discussion to consider general kernel methods by introducing different kernels, different learning tasks, and subspace methods such as kernel PCA. The emphasis is on the flexibility of the approach in applying the analyses to different data, with the caveat that the design of the kernel must rely on domain knowledge. Nonetheless we will argue that, ignoring the technical requirement of positive semi-definiteness, kernel design is not an unnatural task for a practitioner. The overall aim is to give a view of the subject that will enable newcomers to the field to gain their bearings so that they can move to apply or develop the techniques for their particular application.

An Introduction to Support Vector Machines and Other Kernel-based Learning Methods: Background Mathematics

From the publisher: This is the first comprehensive introduction to Support Vector Machines (SVMs), a new generation learning system based on recent advances in statistical learning theory. SVMs deliver state-of-the-art performance in real-world applications such as text categorisation, hand-written character recognition, image classification, biosequences analysis, etc., and are now established as one of the standard tools for machine learning and data mining. Students will find the book both stimulating and accessible, while practitioners will be guided smoothly through the material required for a good grasp of the theory and its applications. The concepts are introduced gradually in accessible and self-contained stages, while the presentation is rigorous and thorough. Pointers to relevant literature and web sites containing software ensure that it forms an ideal starting point for further study. Equally, the book and its associated web site will guide practitioners to updated literature, new applications, and on-line software.